Customize your job with advanced options

Adding environment variables and secrets

The Environment Variables and Secrets concept pages cover how to create and use them. Here we just show the usage for quick reference

from servicefoundry import Job, Build, PythonBuild

job = Job(
    name="iris-train-job",
    image=Build(
        build_spec=PythonBuild(
            command="python train.py",
            requirements_path="requirements.txt",
        )
    ),
    env={
        "DB_USER": "postgres",
        "DB_PASSWORD": "<YOUR_SECRET_FQN>"
    }
)

name: iris-train-job
components:
- name: iris-train-job
  type: job
  image:
    type: build
    build_source:
      type: local
    build_spec:
      type: tfy-python-buildpack
      command: python train.py
      requirements_path: requirements.txt
  env:
    DB_USER: "postgres"
    DB_PASSWORD: "tfy-secret://<YOUR_SECRET_FQN>"

Configure retry limit

You can specify the maximum number of attempts to run a Job before it is marked as failed. By default this is set to 1

from servicefoundry import Job, Build, PythonBuild

job = Job(
    name="iris-train-job",
    image=Build(
        build_spec=PythonBuild(
            command="python train.py",
            requirements_path="requirements.txt",
        )
    ),
    retries=3
)

name: iris-train-job
components:
- name: iris-train-job
  type: job
  image:
    type: build
    build_source:
      type: local
    build_spec:
      type: tfy-python-buildpack
      command: python train.py
      requirements_path: requirements.txt
  retries: 3

Configure Job Time Limit

You can specify (in seconds) the maximum amount of time for a job to run, whether it has failed or not. This will take precedence over the retries Retry Limit. By default this is set to 1000 seconds.

For example, if you set retries to 6 and a timeout of 480 seconds, the job will terminate after 8 minutes regardless of how many times it attempted to run.

from servicefoundry import Job, Build, PythonBuild

job = Job(
    name="iris-train-job",
    image=Build(
        build_spec=PythonBuild(
            command="python train.py",
            requirements_path="requirements.txt",
        )
    ),
    retries=6,
    timeout=480,
)

name: iris-train-job
components:
- name: iris-train-job
  type: job
  image:
    type: build
    build_source:
      type: local
    build_spec:
      type: tfy-python-buildpack
      command: python train.py
      requirements_path: requirements.txt
  retries: 6
  timeout: 480

Set resources limits

You can configure the CPU and Memory resources to be allocated to each job. To understand the resources configuration in more details, please read the Resources concepts page.

For e.g. here we request 0.2 CPU and 128 MiB memory with a hard limit of 0.5 CPU and 512 MiB memory.

from servicefoundry import Job, Build, PythonBuild, Resources

job = Job(
    name="iris-train-job",
    image=Build(
        build_spec=PythonBuild(
            command="python train.py",
            requirements_path="requirements.txt",
        )
    ),
    resource=Resources(
        cpu_request=0.2,
        cpu_limit=0.5,
        memory_request=128,
        memory_limit=512,
    ),
)

name: iris-train-job
components:
- name: iris-train-job
  type: job
  image:
    type: build
    build_source:
      type: local
    build_spec:
      type: tfy-python-buildpack
      command: python train.py
      requirements_path: requirements.txt
  resources:
    cpu_request: 0.2
    cpu_limit: 0.5
    memory_request: 128
    memory_limit: 512

Triggering the job programmatically

Added in servicefoundry>=0.8.0rc1

You can trigger your job programmatically by using thetrigger_job function in Python. Example:

from servicefoundry import trigger_job

trigger_job(
  application_fqn="tfy-ctl-euew1-devtest:tfy-demo:iris-train-job", 
  command="python train.py"
)

`servicefoundry.trigger_job`

Arguments:

application_fqn (str): Fully Qualified Name of the Deployed Job (without the version number)
command(Optional[Union[str, Sequence[str]]]): Command to run the job with, defaults to None. Can be a str or List[str]. When None, the job is triggered with configured command at the time of deployment. When passed as a list, the command will be joined using theshlex.join function.