We don't want to handle any kind of files inside the process engine. If we want to send or receive files over integration artifacts, we have to handle them inside the integration artifact.

We need this kind of file handling (incoming & outgoing) in many integration artifacts (Mail, JMS, DMS, PDF Generation, ...). Makes it sense to create for both ways a Spring Boot starter?

@dominikhorn93 @boal

File Handling in generated Forms

Save File(s)

• There is a 1:n relation between a refId and a file
  • it can be 1:1 but it does not have to
• There is not a direct link between a process instance and a refId
  • It is possible to save all files to one refId but it does not have to be that way
• The cleaning up of the files must be modeled via a template. As input the field keys are needed, in which the RefIds are stored
  • We can easily transfer files between processes
  • We have different expiration dates for different RefIDs
• Authorization is ensured via the processing of a task

So it is currently not possible to use a fileupload at process startup over a generated web form. We need a persistent process start context for this (per user). Of course files can be available if a process is started by an integration artifact like mail or jms.

One challenge is, that we have to know in the task service which S3 Service we want to use for a user task.

Here is the same picture in an asynchronous context:

Load File(s)

Handling

Handling of files over S3 storages is currently restricted to the DigiWF specific storage. Its not possible to handle files in project specific storages.

Within a project specific process, it should be possible to handle files with project specific S3 storages.

Authentication/authorization

All project specific S3 storages expose REST endpoints which will be secured with project specific ROLEs/RIGHTs using Oauth2.

I.e. the call to project specific S3 storages in the Cosys/Mail integration will be handled by Oauth2 client credentials flow. For each call to a project-specific S3 storage, the corresponding ClientID and the corresponding ClientSecret are then required in the Cosys/Mail integration.

Currently, this cannot be handled in the Cosys and Mail services. As a result the current images cannot be used for this purpose.

Concept

• Every process can use a specific S3 service (the s3 service is configured in the process config)
• Every S3 service has its own kafka topic (developers of the S3 service are responsible to claim the topic)

Todos

DigiWF Engine

• Save process config as isntance variable when starting a process
  • adjust processConfigFunction bean to use variable as default and load from db as fallback
• Adjust Task Listener to extract specific s3 from process config and add it as input parameter
• Adjust Rest-Template to add Token by default
• Adjust TaskFileService to use specific S3 URL
• Adjust ProcessDefintionStart Service to use specific S3 URL

S3 Service

• Add async interface for creating presigned urls
• Provide element templates for creating presigned urls

Mail Service

• Adjust async interface to expect a list of presigned urls for loading documents
• adjust element template and add own template process

Cosys Service

• Adjust async interface to expect a presigned url for saving documents
• adjust element template and add own template process

As a process engineer I want to be able to generically start any process via a CIT Web Form

Description

We need an integration artifact to connect to the CIT Form Server.

The CIT integration artifact should be able to

• receive CIT Form submissions in ActiveMQ
• transform the CIT XML structure into JSON
• Upload files to S3
• Start process instances
• Configure relevant form-keys that should start processes

What is not part of this base artifact (digiwf-cit-integration-starter)?

• custom XML to JSON transformation: This should be done in the CIT space via XSLT. An alternative could be to create a custom CIT integration and transform it within that integration artifact.

Details

Receive CIT Form submissions in ActiveMQ

User the CIT ActiveMQ starter to create a base project with the following structure:

• digiwf-cit-integration
• digiwf-cit-integration-starter
• digiwf-cit-integration-service Setup the GitHub Action pipelines and use the oss template.

Transform the CIT XML structure into JSON

Use jackson to automaticaly map XML to JSON (StartInstanceTOV01)

Upload files to S3

upload files to a generated fileConext(UUID) and the corresponding Json Pointer. The functionalty to indentify the correct Json Pointer path should be added to the digiwf-json-serialization starter.

Start process instances

Add a spring cloud stream function that can send a StartInstanceTOV01 to kafka. Add this functionality to the digiwf-cloud-stream-utils library.

Configure relevant form-keys that should start processes

Add the possibility to configure form keys via ENV varibales that can start processes.

Test setup for AMQ

Create a test setup for AMQ and the integration artifact that does no depend on the CIT form server. User the parkausweismanagement XML structure for testing (first customer of that functionality)

As a integration service developer I want to be able to send technical and business errors to the engine. As a process modeler I want to react on error in the model.

Description

The digiwf engine communicates with integration artifacts via asynchronous events sent through Kafka. If an error appears during event processing in the integration artifact, an error event is sent to a configured dead letter queue (dlq). At the moment the errors in the dlq topic are not handled. This behaviour is not ideal, as errors are not detected. To improve the error handling of async events we should listen to error events sent to the dlq and handle them properly in digiWF engine.

Estimation: 20 Days

Details

Digiwf-integration-errors topics

Request a digiwf-integration-errors topic that can be used as a dlq for integration artefacts. Authorize kafka users for the topic

DTO for creating incidents

For creating incidents it is necessary to know in which context the incident should be created. The Incident should be created on the receive task / message event in the bpmn process. to be more precise, in the corresponding job of the message. To query for the message we need the same parameters as we need for correlating. For the first version we can restrict it to:

• messageName: string
• processInstanceId: string
• errorMessage: string

spring cloud stream Incident handler

Create a handler for the header type createIncident The should query for the corresponding exectuion with this.runtimeService.createEventSubscriptionQuery() and use this.runtimeService.createIncident() for creating an incident.

DTO for creating bpmn error

For creating incidents we it is necessary to know in which context the bpmn error should be created.

• processInstanceId: string
• errorCode: string
• errorMessage: string

spring cloud stream bpmn error handler

Create a handler for the header type createBpmnError Check alternatives for correlating messages Correlate the message with the parameters that determine that it is a bpmn error

• errorCode: string
• errorMessage: string

enable bpmn errors by input variable

Only throw bpmn errors if the variable app_bpmn_erorrs_enabled is activated (true)

create example process

Create an example process for incidents and bpmn errors

extend spring cloud stream utils

Extend the spring cloud stream utils for throwing bpmn errors and creating incidents

Description

The process engineer can determine under which S3 path the documents are stored. The previous HumanTaskFileRestController API has the following parameters:

• taskId
• fieldKey

This should be changed to:

• taskId
• filePath

The same applies to the ServiceStartFileRestController. This is to be changed from

• definitionKey
• fieldKey

to

• definitionKey
• filePath

In addition, the AbstractFileService no longer uses the FieldKey to check whether a user has read or write permissions.

Instead the Humtask checks if the corresponding input parameters are set:

• app_file_paths
• app_file_paths_readonly

In the variables there is a ; separated list of paths which may be accessed. For the process start the same parameters are stored in the processconfig.

acceptance criteria

• HumanTaskFile API adjusted
• ProcessStartAPI adjusted
• Generic Json Schema checks in AbstractFileService removed
• ProcessConfig helper Methods created

Description

We need a consumer that listens to BPMN errors and correlates the bpmnError Message in the specific StreamingTemplate process instance.

Details

Request a digiwf-integration-errors topic

• A new event type createBpmnErrorshould be created and added to the typeMappings
• Create the BpmnErrorEventDto
  • processInstanceId: string
  • errorCode: string
  • errorMessage: string
• Correlate the bpmnError Message for the specific processInstanceId and pass the errorCode and errorMessage
• Log.error if a bpmn error cannot be correlated do not create a dlq of bpmn errors

Technical hints

• create a new errors package in the digiwf-connector-core module. You can copy the message package. It will be almost the same
• create a ErrorService Interface that is implemented in digiwf-camunda-connector module. You can almost copy everything from the MessageServiceImpl because we will implement BPMN Errors via a EventSubprocess that is correlated with a message
• extends the *-rest module to easily test the whole setup without the need of a sample integrations service that throws errors. You can copy the message package

Is your feature request related to a problem? Please describe. There is no authorization check when accessing files of a specific task field.

Describe the solution you'd like Check if the user has access to the corresponding field key

Als Entwickler möchte ich die Code Coverage im PR sehen und ein Quality Gate bei 60-80% (new Code) dafür einrichten können.

• JaCoCo in Java/Maven
• Über CodeCov im PR sichtbar machen

Take SysSpec, remove unnecessary chapters, compare result with arc24 and add missing chapters

Beide erscheinen für eine öffentliche Doku etwas überfrachtet und enthalten viele Kapitel, die eher intern ins Betriebshandbuch gehören. Daher hier ein Versuch der Reduktion:

1. Einführung
2. IT-Architektur
3. Querschnittliche Konzepte
4. Schnittstellen
5. Entwurfsentscheidungen

Vielleicht starten wir erstmal eher mit weniger und nehmen Kapitel hinzu, wenn diese uns fehlen.

Das Ziel der TechDoku als UserStory formuliert: Als OpenSource Entwickler*in für die Plattform möchte ich das Gesamtsystem verstehen und Entscheidungen nachvollziehen können, sodass ich schnell einen Überblick bekomme aber auch tiefgreifende Konzepte nachvollziehen kann und damit mich an der Entwicklung der Plattform beteiligen kann.

Add internal digiwf-docs to monorepo and build a pipeline to publish the documentation

acceptance criteria:

• base structure for documentation is in monorepo
• documentation is up and running

Is your feature request related to a problem? Please describe.

The digiwf tasklist loads all user/group tasks assigned to the user at once. If a user has multiple hundred or thousand tasks assigned, it takes a while for the tasklist to load all tasks from the api.

Describe the solution you'd like

We should refactore the frontend and the task api to not load all tasks at once. Maybe we use api pagination (backend) or another concept to reduce the amount of data.

Describe alternatives you've considered

Alternatively, we could move to a more real time functionality by using websockets and/or graphql to stream the task data to the frontend.

Additional context

Loadtest results: https://git.muenchen.de/digitalisierung/digiwf-load-test/-/pipelines/125870

Priority

Go live of parkausweis managment is in november. Therefore, it would be great if we could fix this issue advance.

Bumps eslint from 8.23.1 to 8.31.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps core-js from 3.25.1 to 3.27.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps marked from 4.1.0 to 4.2.5.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Currently incidents notifications from prod (mails) are not send or have the wrong env declaration. We need to check this.

Are E-Mail Notifications send from Prod and which Env-Title do they have?

https://github.com/it-at-m/digiwf-core/blob/1c4ca8303ae95c8cf4ddc3e529906ea0d2a53fa2/digiwf-engine/digiwf-engine-service/src/main/java/io/muenchendigital/digiwf/engine/incidents/IncidentNotifierHandler.java