Using a managed object storage service (S3 or GCS)

Object storage is used by various Sourcegraph features, for example to store code graph indexes uploaded by users or the results of search jobs.

By default, Sourcegraph will use a sourcegraph/blobstore server bundled with the instance. This is intended as a temporary measure: we recommend that administrators configure self-hosted Sourcegraph to store this data in an AWS S3 or Google Cloud Storage (GCS) bucket following the guidance below. Doing so may decrease your hosting costs as persistent volumes are often more expensive than the same storage space in an object store service.

Sourcegraph bucket

On S3 or GCS, the Sourcegraph bucket is the single bucket for Sourcegraph features: each feature stores its objects under its own key prefix within this bucket. This includes code graph indexes and search job results, unless they have legacy configuration that uses S3 or GCS.

Configure frontend

Set the SOURCEGRAPH_UPLOAD_* environment variables described below on the frontend service. Frontend distributes this configuration to the other services that use the bucket: worker, precise Code Intel (precise-code-intel-worker), syntactic Code Intel (syntactic-code-intel-worker), gitserver, and searcher.

• A service that sets SOURCEGRAPH_UPLOAD_BACKEND reads all SOURCEGRAPH_UPLOAD_* variables from its own environment instead of using frontend's configuration, so configuration set on each service keeps working. To use frontend's configuration, remove the SOURCEGRAPH_UPLOAD_* variables from the other services. A service that doesn't set SOURCEGRAPH_UPLOAD_BACKEND ignores its other SOURCEGRAPH_UPLOAD_* variables.
• Frontend distributes only static credentials: AWS access keys and GCS credentials file contents. Every service that uses the bucket resolves other credentials itself, as described for GCS and S3.
• When frontend's configuration changes, the services that use it restart on their own to apply it.

Using GCS for the Sourcegraph bucket

Set the following environment variables to target a GCS bucket for shared Sourcegraph uploads.

• SOURCEGRAPH_UPLOAD_BACKEND=GCS
• SOURCEGRAPH_UPLOAD_BUCKET=<my bucket name>
• SOURCEGRAPH_UPLOAD_GCP_PROJECT_ID=<my project id> (optional; required with SOURCEGRAPH_UPLOAD_MANAGE_BUCKET=true)
• SOURCEGRAPH_UPLOAD_GOOGLE_APPLICATION_CREDENTIALS_FILE=</path/to/file> (optional)
• SOURCEGRAPH_UPLOAD_GOOGLE_APPLICATION_CREDENTIALS_FILE_CONTENT=<{"my": "content"}> (optional)

Frontend distributes the path in SOURCEGRAPH_UPLOAD_GOOGLE_APPLICATION_CREDENTIALS_FILE, not the file. If you set it, mount the file at the same path in every service that uses the bucket.

If you are running on GKE with Workload Identity, or otherwise relying on Application Default Credentials, you can omit the GCS credentials file variables. Grant the roles/storage.objectAdmin role to the service account of every service that uses the bucket.

Using S3 for the Sourcegraph bucket

Set the following environment variables to target an S3 bucket for shared Sourcegraph uploads.

• SOURCEGRAPH_UPLOAD_BACKEND=S3
• SOURCEGRAPH_UPLOAD_BUCKET=<my bucket name>
• SOURCEGRAPH_UPLOAD_AWS_REGION=us-east-1
• SOURCEGRAPH_UPLOAD_AWS_ENDPOINT=https://s3.us-east-1.amazonaws.com
• SOURCEGRAPH_UPLOAD_AWS_ACCESS_KEY_ID=<your access key>
• SOURCEGRAPH_UPLOAD_AWS_SECRET_ACCESS_KEY=<your secret key>
• SOURCEGRAPH_UPLOAD_AWS_SESSION_TOKEN=<your session token> (optional)
• SOURCEGRAPH_UPLOAD_AWS_USE_EC2_ROLE_CREDENTIALS=true (optional; set to use EC2 metadata API over static credentials)
• SOURCEGRAPH_UPLOAD_AWS_USE_PATH_STYLE=false (optional)

You don't need to set the SOURCEGRAPH_UPLOAD_AWS_ACCESS_KEY_ID environment variable when using SOURCEGRAPH_UPLOAD_AWS_USE_EC2_ROLE_CREDENTIALS=true because role credentials will be automatically resolved by each service. Grant the IAM role to every service that uses the bucket, for example by attaching it to the EC2 instances hosting those containers in a multi-node environment.

Automatically provision the Sourcegraph bucket

Most deployments should provision this bucket directly in their cloud provider and leave this disabled. If you would like to allow your Sourcegraph instance to manage the target bucket configuration, set the following environment variable:

• SOURCEGRAPH_UPLOAD_MANAGE_BUCKET=true

Legacy per-feature buckets

Code graph index uploads and Search Job results can also be stored in their own buckets. These per-feature buckets predate the Sourcegraph bucket, which Sourcegraph 7.2 introduced. Instances that don't already use them only need the Sourcegraph bucket.

A feature keeps using its own bucket if its BACKEND variable, such as PRECISE_CODE_INTEL_UPLOAD_BACKEND, is S3 or GCS. Otherwise, the feature uses the Sourcegraph bucket. When the Sourcegraph bucket is on S3 or GCS, the feature's objects are under a key prefix named after its BUCKET variable: lsif-uploads/ or search-jobs/ by default.

The legacy variables have the same names and meanings as the Sourcegraph bucket variables, with the feature's prefix in place of SOURCEGRAPH_UPLOAD_: for example, PRECISE_CODE_INTEL_UPLOAD_BUCKET in place of SOURCEGRAPH_UPLOAD_BUCKET. With GCS, the feature's GCP_PROJECT_ID variable is required.

Frontend doesn't distribute legacy configuration: set the variables on every service listed for the feature, and grant those services access to the bucket when they use Application Default Credentials or IAM roles.