Master thesis: Controlling industrial robots with gener-ative AI

Webseite Fraunhofer IEM

Wie sieht das Engineering der Zukunft aus? Zu dieser Frage entwickelt das Fraunhofer IEM in Paderborn überzeugende Lösungen – von der Geschäftsidee über die Umsetzung bis zum Markterfolg. Im Fokus stehen intelligente Produkte, Produktionssysteme, Dienstleistungen und Softwareanwendungen. Unsere Wissenschaftlerinnen und Wissenschaftler arbeiten interdisziplinär an neuen Methoden, Werkzeugen sowie Prozessen und setzen innovative Technologien ein, um die Wettbewerbsfähigkeit unserer Kunden und Partner langfristig zu sichern.

Problem and objective

Large Language Models (LLMs) are experienc-
ing a significant surge in popularity. Following

the release of ChatGPT, their adoption has dra-
matically increased, with a broad range of users

reporting widespread usage and practical ben-
efits. By leveraging Generative Artificial Intelli-
gence (GenAI) in work environments, productiv-
ity can be enhanced, and results improved.

While most tasks utilize LLMs‘ extensive gen-
eral knowledge, these models can be fine-tuned

to understand and solve complex problems. Ex-
amples include safety monitoring during pro-
duction or generative design for improved ma-
terial usage. Therefore, LLMs and GenAI have

the potential to revolutionize entire industries.
LLMs can generate multiple types of content,
such as images, text, or source code. A notable
example of the utilization of LLMs for source
code generation is the GitHub Copilot, which

significantly impacted the field of software de-
velopment.

Source code generation can also be used to
control industrial robots, like ABBs YuMi series.

Currently, programming such robots requires
expert knowledge of the robot’s mechanics and
a translation of the desired movements into
code. This creates a barrier to using robots due

to high training and implementation costs. Con-
trolling the robot via natural language would sig-
nificantly lower this barrier. However, it remains

unclear whether and how LLMs can effectively
solve this problem.
To address this uncertainty, the main objective
of this thesis is to develop a concept for robot

control using LLMs to translate natural lan-
guage inputs into suitable code for the robot.

This includes a comprehensive analysis of ex-
isting LLM-based approaches for source code

generation, evaluating the capabilities of LLMs
for robot control, developing a concept for robot

control using LLMs, designing a reference ar-
chitecture, and implementing a proof of con-
cept. Furthermore, the developed approach will

be generalized and abstracted into a generic

methodology that can be reused for future pro-
jects.

Work program
1. Familiarization with the task and
preparation of a detailed outline

130 h

2. Problem analysis to determine re-
quirements for the approach

150 h
3. Analysis of the state of the art 120 h
5. Development and validation of a
systematic approach
• Identification of relevant
communication interfaces
• Selection of suitable training
data
• Tuning of the selected LLM

• Implementation and valida-
tion of the approach

200 h

6. Documentation of the results 130 h
7. Final thesis presentation 20 h
750 h

Remarks
The times planned in the work program are
standard values.

Supervision
M.Sc. Benjamin Tiggemann
M.Sc. Ruslan Bernijazov
Heinz Nixdorf Institut
Universität Paderborn
Fürstenallee 11, 33102 Paderborn
E-Mail: Benjamin.Tiggemann@hni.upb.de
E-Mail: Ruslan.Bernijazov@hni.upb.de

Um sich für diesen Job zu bewerben, sende deine Unterlagen per E-Mail an Benjamin.Tiggemann@hni.upb.de