Set up Python local development

It is possible to carry out local development of Transforms Python repos, allowing for high-speed iterative development.

Setting up local development for Python transforms repositories

Clone the repository

  1. In the menu bar of your repository, select Work locally to open the dialog and copy the given repository URL.

The top menu bar of a repository with the "Work locally" option to the right.

The "Work locally" dialog.

  1. Using the command line, run git clone <URI> on your local machine in a directory of your choice. Then use the cd command to navigate to the repository.

Limitations

  • The token granted for cloning is short-lived and read-only, with the exception of pushing back to your repository.
  • You will still need to push your changes to Foundry to publish job specs or artifacts, or if you wish to run checks or build.

Preview

Dataset Previews are supported in local development. See Local preview for more details.

Set up the development environment

Prerequisites

  • Ensure Java 17 is installed and that the environment variable JAVA_HOME points to the right Java installation. Java 17 can be downloaded from the Oracle website ↗.

Setting the JAVA_HOME environment variable based on your operating system:

  • Windows: Run SETX JAVA_HOME -m "<java-home-dir>" in PowerShell. This modifies the system environment variable and you will need to restart the shell for changes to take effect. Alternatively you can run [System.Environment]::SetEnvironmentVariable("JAVA_HOME", "<java-home-dir>") to set JAVA_HOME in the running process.
  • Linux or macOS: Run export JAVA_HOME=<java-home-dir>.
  • Ensure your repository is upgraded to the latest template version by following the steps outline here.

  • Ensure that the environment variables CI, JEMMA, and CA are not set.

  • If running on an Apple silicon Mac, ensure that Rosetta 2 ↗ is installed. You can install Rosetta 2 by running /usr/sbin/softwareupdate --install-rosetta --agree-to-license in the terminal.

Visual Studio Code

Requirements

Beta

The Palantir extension for Visual Studio Code is in a beta state and may not be available for use in your enrollment.

To use the Palantir extension for Visual Studio Code, you must have access to a code repository in your enrollment.

If you see unexpected language server errors, such as broken imports, it may be that the automatic set up of the Python interpreter failed. To fix this you can manually set up the interpreter by navigating to the Command Palette ↗, then typing Python: Select interpreter and choosing a Python interpreter matching the path: ./maestro/bin/python. The .maestro folder should be present if the Palantir extension successfully ran the command Palantir: Install Python environment.

PyCharm

  • To set up a Python development environment, run the command ./gradlew condaDevelop.

  • Ensure you have IntelliJ PyCharm ↗ installed locally.

  • Import the project following the steps outlined here ↗.

  • Choose Add New Interpreter from the Python Interpreter selector ↗ on the status bar.

    add python interpreter screenshot

  • In the left-hand pane of the Add Python Interpreter dialog, select Virtualenv Environment.

    configure python interpreter screenshot

  • Choose Existing environment and set the Interpreter field to the Python interpreter from your Conda environment. The Conda enviroment path can be listed by running ./gradlew condaEnvList.

    • For Unix, the Python interpreter path is <your-conda-environment-dir>/bin/python.
    • For Windows, the Python interpreter path is <your-conda-environment-dir>\python.exe.

Depending on whether the test plugin is enabled, the installed environment list would include either ./transforms-python/build/conda/run-env or ./transforms-python/build/conda/test-env or both. You should pick the test environment if you plan on running tests.

  • Select Ok.

(Advanced setup) Gradle-based environment

If the Palantir extension for Visual Studio Code is not available in your enrollment, it is possible to use Gradle to set up your Python environment. In a new terminal window, navigate to the repository directory using the cd command, then run ./gradlew vsCode. This will create a Python environment and configure Visual Studio Code to use the environment's Python interpreter by placing it in the .vscode/setting.json file.