Batch App

This section describes how to create a Batch Carol App from Carol's template.

To start the process to create a Carol App, you need to run the following command:

bash -c "$(curl -fsSL https://raw.github.com/totvslabs/carolapp/master/install.sh)"

Batch Carol App

After selecting to create a Batch Carol App, the following question may appear in case this is not your first time running this process:

19021902

You can press enter, sending yes, for this process to download the files again to make sure you are running the most updated templates.

After that, the whole process will ask you the following questions:

full_name [Totvs Labs]: Robson Poffo

email [[email protected]]: [email protected]

github_username [totvslabs]:

project_name [Carol App Batch Boilerplate]: First Batch App

project_slug [first_batch_app]:

project_short_description [Carol App Batch Boilerplate contains all the boilerplate you need to create a Carol App.]: This is the description related to my project. Used inside the manifest file.

version [0.1.0]:

sonarqube [True]:

carol_app_organization [my_organization]: poffo

carol_app_tenant [my_tenant]: poffo

algorithm_name [first_batch_app_algorithm]: process_data

algorithm_description_en_US [Batch task Description]: Process responsible to read data from Carol, run the processing it is responsible for and send the final result back to Carol.

algorithm_description_pt_BR [Descrição do Tarefa Batch]: Processo responsavel por ler os dados da Carol, executar os procedimentos no qual ele se propos a fazer e enviar o resultado final para Carol.

luigi_workers [4]:

carol_app_docker_image [carol/first_batch_app:0.1.0]:

pycarol_version [latest]:

Select carol_app_machine_types:
1 - c1.nano
2 - c1.micro
3 - c1.small
4 - c1.medium
5 - c1.large
6 - c1.xlarge
7 - c1.2xlarge
8 - g1.v100.large
9 - g1.v100.xlarge
Choose from 1, 2, 3, 4, 5, 6, 7, 8, 9 (1, 2, 3, 4, 5, 6, 7, 8, 9) [1]: 1
First Batch App created successfully.

✔  Your new Carol App was successfull created. Check Carol App documentation to learn more.

This is the brief description for each parameter requested:

PropertyDescription
full_nameThe user creating this application. This information is used for internal purposes.
emailThe user creating this application. This information is used for internal purposes.
github_usernameThe user creating this application. This information is used for internal purposes.
project_nameThe project name is used to identify the project and update the Readme.md file.
project_slugSlug project is used to create the folder of your project.
project_short_descriptionIt is used to update the Readme.md file.
versionIt is used for internal reference for this application and to update the Readme.md file.
sonarqubeIn case you enable it, the process will prepare the project to run sonarqube.
carol_app_organizationCarol's organization where this project will be deployed.
carol_app_tenantCarol's tenant where this project will be deployed.
algorithm_nameThe algorithm name is the algorithm that this process will start. Since the Batch Carol App can have multiple processes inside the same application, you can have multiple Batch processes pointing to different algorithms.

For reference related to the manifest file, please refer to the chapter "Manifest file" on this documentation for more details.
algorithm_description_en_USThe algorithm description (US) is used to share information with the user on Carol's UI. This information will be updated in the manifest file.

Refer to the section "Manifest file" for more information about this file.
algorithm_description_pt_BRThe algorithm description (US) is used to share information with the user on Carol's UI. This information will be updated in the manifest file.

Refer to the section "Manifest file" for more information about this file.
luigi_workersThis is the number of workers when running the tasks on Luigi.

For more information, please verify Luigi's documentation related to this topic: https://luigi.readthedocs.io/en/stable/execution_model.html
carol_app_docker_imageThe docker image provided at this point will update the Dockerfile to build the docker image with the specified name and update the manifest file. When you run this application it will look for the specified docker name specified here.
pycarol_versionPyCarol version refers to the pyCarol version this app has the dependency. The file requirements.txt will be updated with this information.
carol_app_machine_typesThis is the instance type to run this application. To verify all possible instance types, please, verify the documentation.

Once you finished the process, you will see the following success message:

13001300

This message means everything is fine. You can enter the new folder created (slug) and we can start exploring the available "make" options.

The following commands are available when we run the command "make".

$ make
first_batch_app Makefile help.

all_tests                      Runs all test related tasks
build_ci                       Build CI Docker Image
clean                          Cleans all temporary and preprocessed files
code_scan                      Execute a Code Qualiry scan and send the data to SonarQube server
coverage                       Create XML and HTML Test coverare report.
dev                            Install all development packages dependencies
dist                           Create the dist file to deploy on Carol
docker_image                   Build the Carol App docker image
docs                           Compile all package documentation in HTML format
help                           Show this help message
htmlcov                        Opens the current test coverage report on the default browser
init                           Install all package dependencies
lint                           Lint all code for best practicies
run                            Runs the Batch process locally
test                           Runs all project test suite, recording all test converage data
venv                           Create the virtualenv for this Carol App

Additional information:

  • Batch Carol App can have multiple processes. During the creation flow, it asks just one, but any time you can go to your manifest file and add more processes as needed.
  • The docker image builds provided by make build the image locally. In the section "Building Docker Image on Carol" we will explain how to build the docker image on the platform, making the docker image available for any Carol App we run on top of the platform Carol.

Building the Docker Image

If you want to test your local image into your own docker registry, you can build the image using the make command, tag your image to prepare it for the push and after, you need to push it into your registry. These are the commands to perform it:

First, we build the image:

make docker_image

This command will return something similar:

.....
Successfully built b0bf90002a7d
Successfully tagged first_batch_app-process_data:0.1.0

Next, we tag the image to prepare it to push to your Docker registry:

docker tag first_batch_app-process_data:0.1.0 robsonpoffo/first_batch_app-process_data:0.1.0

The last step is to push it to your own Docker Registry. If you do not have, please verify the steps here: https://docs.docker.com/registry/

docker push robsonpoffo/first_batch_app-process_data:0.1.0

This command will return something similar:

The push refers to repository [docker.io/robsonpoffo/first_batch_app-process_data]
ddb9ffc5b1b9: Pushed
bbe6a88aa3cf: Pushed
f8be782735d6: Mounted from robsonpoffo/first_batch_app
165b81c874a8: Mounted from robsonpoffo/first_batch_app
e5cf317c6146: Mounted from robsonpoffo/first_online_app
7d1b621fef71: Mounted from robsonpoffo/first_online_app
93a8b7ed8c52: Mounted from robsonpoffo/first_online_app
3f975a33acbf: Mounted from robsonpoffo/first_online_app
69168fb9b7b0: Mounted from robsonpoffo/first_online_app
31f78d833a92: Mounted from robsonpoffo/first_online_app
2ea751c0f96c: Mounted from robsonpoffo/first_online_app
7a435d49206f: Mounted from robsonpoffo/first_online_app
9674e3075904: Mounted from robsonpoffo/first_online_app
831b66a484dc: Mounted from robsonpoffo/first_online_app
0.1.0: digest: sha256:e9b9d1d347ecb4c53b664595b1472c3005c43ce36855e90ce28db3382028fd32 size: 3261

Updating Docker Image on Manifest file

After that, you can update your manifest file and test the Docker Image inside Carol:

{
  "batch": {
    "processes": [
      {
        "algorithmDescription": {
          "en-US": "Process responsible to read data from Carol, run the processing it is responsible for and send the final result back to Carol.",
          "pt-BR": "Processo responsavel por ler os dados da Carol, executar os procedimentos no qual ele se propos a fazer e enviar o resultado final para Carol.",
          "pt-br": "Processo responsavel por ler os dados da Carol, executar os procedimentos no qual ele se propos a fazer e enviar o resultado final para Carol."
        },
        "algorithmName": "process_data",
        "algorithmTitle": {
          "en-US": "Process responsible to read data from Carol, run the processing it is responsible for and send the final result back to Carol.",
          "pt-BR": "Processo responsavel por ler os dados da Carol, executar os procedimentos no qual ele se propos a fazer e enviar o resultado final para Carol."
        },
        "instanceProperties": {
          "tenants": {
            "EXAMPLE_ENV_1": "value_1",
            "EXAMPLE_ENV_2": "value_2"
          },
          "keepInstance": "false",
          "luigi": {
            "workers": "4"
          },
          "profile": "",
          "properties": {
            "dockername": "first_batch_app-process_data"
          }
        },
        "name": "first_batch_app"
      }
    ]
  }
}

Additional information

Deploying the Batch Carol App in Carol

After that, in Carol:

  • Upload the manifest file:
33003300
  • Execute the task:
33063306
  • See that the task was executed successfully:
33023302