Configuration

Processor configuration

Honcho runs a processor process and a disk cache manager process. Both share the same configuration.

Configuration

Configuration summary:

Setting

Parser

Required?

TOOL_ENV

bool

LOCAL_DEV_ENV

bool

HOSTNAME

str

SENTRY_DSN

str

LOGGING_LEVEL

str

STATSD_HOST

str

STATSD_PORT

int

LOCAL_DEV_AWS_ENDPOINT_URL

str

CLOUD_PROVIDER

cloud_provider_parser

CACHE_MANAGER_LOGGING_LEVEL

str

SYMBOLS_CACHE_PATH

str

SYMBOLS_CACHE_MAX_SIZE

or_none(parse_data_size)

BETAVERSIONRULE_VERSION_STRING_API

str

STAGE_SUBMITTER_LOGGING_LEVEL

str

STAGE_SUBMITTER_DESTINATIONS

ListOf(str)

PROCESSOR_TEMPORARY_PATH

str

STACKWALKER_COMMAND_PATH

str

STACKWALKER_KILL_TIMEOUT

or_none(parse_time_period)

STACKWALKER_SYMBOLS_URLS

ListOf(str)

SYMBOLS_TMP_PATH

str

SQS_STANDARD_QUEUE

str

SQS_PRIORITY_QUEUE

str

SQS_REPROCESSING_QUEUE

str

TELEMETRY_S3_ACCESS_KEY

str

TELEMETRY_S3_SECRET_ACCESS_KEY

str

TELEMETRY_S3_REGION

str

PUBSUB_PROJECT_ID

str

PUBSUB_STANDARD_TOPIC_NAME

str

PUBSUB_STANDARD_SUBSCRIPTION_NAME

str

PUBSUB_PRIORITY_TOPIC_NAME

str

PUBSUB_PRIORITY_SUBSCRIPTION_NAME

str

PUBSUB_REPROCESSING_TOPIC_NAME

str

PUBSUB_REPROCESSING_SUBSCRIPTION_NAME

str

TELEMETRY_GCS_BUCKET

str

ELASTICSEARCH_INDEX

str

ELASTICSEARCH_INDEX_REGEX

str

ELASTICSEARCH_URL

str

Yes

PROCESSOR_NUMBER_OF_THREADS

or_none(int)

PROCESSOR_MAXIMUM_QUEUE_SIZE

or_none(int)

Configuration options:

TOOL_ENV
Parser

bool

Default

“False”

Required

No

Whether or not this is running in a tool environment and should ignore required configuration.

LOCAL_DEV_ENV
Parser

bool

Default

“False”

Required

No

Whether or not this is a local development environment.

HOSTNAME
Parser

str

Default

“socket.gethostname()”

Required

No

Name of the host this is running on.

SENTRY_DSN
Parser

str

Default

“”

Required

No

Sentry DSN or empty string to disable.

LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

Default logging level. Should be one of DEBUG, INFO, WARNING, ERROR.

STATSD_HOST
Parser

str

Default

“localhost”

Required

No

statsd host.

STATSD_PORT
Parser

int

Default

“8125”

Required

No

statsd port.

LOCAL_DEV_AWS_ENDPOINT_URL
Parser

str

Default

“”

Required

No

Endpoint url for AWS SQS/S3 in the local dev environment. Don’t set this in server environments.

CLOUD_PROVIDER
Parser

cloud_provider_parser

Default

“AWS”

Required

No

The cloud provider to use for queueing and blob storage. Must be AWS or GCP.

CACHE_MANAGER_LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

Default logging level for the cache manager. Should be one of DEBUG, INFO, WARNING, ERROR.

SYMBOLS_CACHE_PATH
Parser

str

Default

“os.path.join(tempfile.gettempdir(), “symbols”, “cache”)”

Required

No

Directory to use for the on-disk LRU-cache for symbols files.

SYMBOLS_CACHE_MAX_SIZE
Parser

or_none(parse_data_size)

Default

“40gb”

Required

No

Max size (bytes) of symbols cache. You can use _ to group digits for legibility. You can use units like kb, mb, gb, tb, etc.

BETAVERSIONRULE_VERSION_STRING_API
Parser

str

Default

https://crash-stats.mozilla.org/api/VersionString

Required

No

URL for the version string API endpoint in the Crash Stats webapp.

STAGE_SUBMITTER_LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

Default logging level for the stage submitter. Should be one of DEBUG, INFO, WARNING, ERROR.

STAGE_SUBMITTER_DESTINATIONS
Parser

ListOf(str)

Default

“”

Required

No

Comma-separated pairs of DESTINATION_URL|SAMPLE where the DESTINATION_URL is an https url to submit the crash report to and SAMPLE is a number between 0 and 100 representing the sample rate. For example:

  • https://example.com|20

  • https://example.com|30,https://example2.com|100

PROCESSOR_TEMPORARY_PATH
Parser

str

Default

“tempfile.gettempdir()”

Required

No

Directory to use as a workspace for crash report processing.

STACKWALKER_COMMAND_PATH
Parser

str

Default

“/stackwalk-rust/minidump-stackwalk”

Required

No

Aboslute path to the stackwalker binary.

STACKWALKER_KILL_TIMEOUT
Parser

or_none(parse_time_period)

Default

“2m”

Required

No

Timeout in seconds before the stackwalker is killed.

STACKWALKER_SYMBOLS_URLS
Parser

ListOf(str)

Default

“”

Required

No

Comma-separated list of urls for symbols suppliers.

SYMBOLS_TMP_PATH
Parser

str

Default

“os.path.join(tempfile.gettempdir(), “symbols”, “tmp”)”

Required

No

Directory to use for temporary storage of files being downloaded.

SQS_STANDARD_QUEUE
Parser

str

Default

“standard-queue”

Required

No

Name for the standard processing queue.

SQS_PRIORITY_QUEUE
Parser

str

Default

“priority-queue”

Required

No

Name for the priority processing queue.

SQS_REPROCESSING_QUEUE
Parser

str

Default

“reprocessing-queue”

Required

No

Name for the reprocessing queue.

TELEMETRY_S3_ACCESS_KEY
Parser

str

Default

“”

Required

No

S3 access key for telemetry bucket.

TELEMETRY_S3_SECRET_ACCESS_KEY
Parser

str

Default

“”

Required

No

S3 secret access key for telemetry bucket.

TELEMETRY_S3_REGION
Parser

str

Default

“”

Required

No

S3 region.

PUBSUB_PROJECT_ID
Parser

str

Default

“test”

Required

No

Google Compute Platform project_id.

PUBSUB_STANDARD_TOPIC_NAME
Parser

str

Default

“standard-queue”

Required

No

Topic name for the standard processing queue.

PUBSUB_STANDARD_SUBSCRIPTION_NAME
Parser

str

Default

“standard-queue”

Required

No

Subscription name for the standard processing queue.

PUBSUB_PRIORITY_TOPIC_NAME
Parser

str

Default

“priority-queue”

Required

No

Topic name for the priority processing queue.

PUBSUB_PRIORITY_SUBSCRIPTION_NAME
Parser

str

Default

“priority-queue”

Required

No

Subscription name for the priority processing queue.

PUBSUB_REPROCESSING_TOPIC_NAME
Parser

str

Default

“reprocessing-queue”

Required

No

Topic name for the reprocessing queue.

PUBSUB_REPROCESSING_SUBSCRIPTION_NAME
Parser

str

Default

“reprocessing-queue”

Required

No

Subscription name for the reprocessing queue.

TELEMETRY_GCS_BUCKET
Parser

str

Default

“”

Required

No

GCS bucket name for telemetry data export.

ELASTICSEARCH_INDEX
Parser

str

Default

“socorro%Y%W”

Required

No

Template for Elasticsearch index names.

ELASTICSEARCH_INDEX_REGEX
Parser

str

Default

“^socorro[0-9]{6}$”

Required

No

Regex for matching Elasticsearch index names.

ELASTICSEARCH_URL
Parser

str

Required

Yes

Elasticsearch url.

PROCESSOR_NUMBER_OF_THREADS
Parser

or_none(int)

Default

“4”

Required

No

Number of worker threads for the processor.

PROCESSOR_MAXIMUM_QUEUE_SIZE
Parser

or_none(int)

Default

“8”

Required

No

Number of items to queue up from the processing queues.

Webapp / crontabber configuration

Gunicorn configuration:

GUNICORN_TIMEOUT
Parser

str

Default

“300”

Required

No

Specifies the timeout value in seconds.

https://docs.gunicorn.org/en/stable/settings.html#timeout

Used in bin/run_service_webapp.sh.

GUNICORN_WORKERS
Parser

str

Default

“1”

Required

No

Specifies the number of gunicorn workers.

You should set it to (2 x $num_cores) + 1.

https://docs.gunicorn.org/en/stable/settings.html#workers

http://docs.gunicorn.org/en/stable/design.html#how-many-workers

Used in bin/run_service_webapp.sh.

GUNICORN_WORKER_CLASS
Parser

str

Default

“sync”

Required

No

Specifies the gunicorn worker type.

https://docs.gunicorn.org/en/stable/settings.html#workers

Used in bin/run_service_webapp.sh.

GUNICORN_MAX_REQUESTS
Parser

str

Default

“10000”

Required

No

The number of requests before recycling the gunicorn worker.

https://docs.gunicorn.org/en/stable/settings.html#workers

Used in bin/run_service_webapp.sh.

GUNICORN_MAX_REQUESTS_JITTER
Parser

str

Default

“1000”

Required

No

The range to generate a random amount to add to max requests so that everything isn’t restarting at the same time.

https://docs.gunicorn.org/en/stable/settings.html#workers

Used in bin/run_service_webapp.sh.

Webapp configuration:

Configuration

Configuration summary:

Setting

Parser

Required?

TOOL_ENV

parse_bool

LOCAL_DEV_ENV

parse_bool

DEBUG

parse_bool

STATIC_ROOT

str

ALLOWED_HOSTS

ListOf(str)

LOGGING_LEVEL

str

DJANGO_LOGGING_LEVEL

str

HOSTNAME

str

BZAPI_BASE_URL

str

BZAPI_TOKEN

str

CACHE_IMPLEMENTATION_FETCHES

parse_bool

STATSD_HOST

str

STATSD_PORT

int

TIME_ZONE

str

DATABASE_URL

str

NPM_ROOT_PATH

str

SECRET_KEY

str

Yes

SESSION_COOKIE_SECURE

parse_bool

SESSION_COOKIE_HTTPONLY

parse_bool

X_FRAME_OPTIONS

str

OVERVIEW_VERSION_URLS

str

SENTRY_DSN

str

ANALYZE_MODEL_FETCHES

parse_bool

OIDC_RP_CLIENT_ID

str

OIDC_RP_CLIENT_SECRET

str

OIDC_OP_AUTHORIZATION_ENDPOINT

str

OIDC_OP_TOKEN_ENDPOINT

str

OIDC_OP_USER_ENDPOINT

str

LAST_LOGIN_MAX

int

NUMBER_OF_FEATURED_VERSIONS

int

VERSIONS_WINDOW_DAYS

int

VERSIONS_COUNT_THRESHOLD

int

LESS_BINARY

str

UGLIFYJS_BINARY

str

CSSMIN_BINARY

str

CACHE_LOCATION

str

CACHE_TIMEOUT

int

CACHE_KEY_PREFIX

str

Configuration options:

TOOL_ENV
Parser

parse_bool

Default

“false”

Required

No

Whether or not we’re running in a tool environment where we want to ignore required configuration

LOCAL_DEV_ENV
Parser

parse_bool

Default

“false”

Required

No

Whether or not we’re running in the local development environment

DEBUG
Parser

parse_bool

Default

“false”

Required

No

Debugging displays nice error messages, but leaks memory. Set this to false on all server instances and true only for development.

STATIC_ROOT
Parser

str

Default

“path(“static”)”

Required

No

Absolute path to the directory static files should be collected to.

ALLOWED_HOSTS
Parser

ListOf(str)

Default

“”

Required

No

LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

Logging level for Crash Stats code

DJANGO_LOGGING_LEVEL
Parser

str

Default

“INFO”

Required

No

Logging level for Django logging (requests, SQL, etc)

HOSTNAME
Parser

str

Default

“socket.gethostname()”

Required

No

Name of the host this is running on.

BZAPI_BASE_URL
Parser

str

Default

https://bugzilla.mozilla.org/rest

Required

No

Base URL for Bugzilla API

BZAPI_TOKEN
Parser

str

Default

“”

Required

No

Bugzilla API token

CACHE_IMPLEMENTATION_FETCHES
Parser

parse_bool

Default

“true”

Required

No

STATSD_HOST
Parser

str

Default

“localhost”

Required

No

statsd host.

STATSD_PORT
Parser

int

Default

“8125”

Required

No

statsd port.

TIME_ZONE
Parser

str

Default

“UTC”

Required

No

DATABASE_URL
Parser

str

Default

“sqlite://sqlite.crashstats.db”

Required

No

NPM_ROOT_PATH
Parser

str

Default

“ROOT”

Required

No

SECRET_KEY
Parser

str

Required

Yes

Make this unique, and don’t share it with anybody. It cannot be blank.

Parser

parse_bool

Default

“true”

Required

No

If you intend to run WITHOUT HTTPS, such as local development, then set this to false

Parser

parse_bool

Default

“true”

Required

No

By default, use HTTPONLY cookies

X_FRAME_OPTIONS
Parser

str

Default

“DENY”

Required

No

By default, we don’t want to be inside a frame. If you need to override this you can use the django.views.decorators.clickjacking.xframe_options_sameorigin decorator on specific views that can be in a frame.

OVERVIEW_VERSION_URLS
Parser

str

Default

“”

Required

No

Comma-separated list of urls that serve version information in JSON format

SENTRY_DSN
Parser

str

Default

“”

Required

No

Sentry aggregates reports of uncaught errors and other events

ANALYZE_MODEL_FETCHES
Parser

parse_bool

Default

“true”

Required

No

Set to true enable analysis of all model fetches

OIDC_RP_CLIENT_ID
Parser

str

Default

“”

Required

No

OIDC_RP_CLIENT_SECRET
Parser

str

Default

“”

Required

No

OIDC_OP_AUTHORIZATION_ENDPOINT
Parser

str

Default

“”

Required

No

OIDC_OP_TOKEN_ENDPOINT
Parser

str

Default

“”

Required

No

OIDC_OP_USER_ENDPOINT
Parser

str

Default

“”

Required

No

LAST_LOGIN_MAX
Parser

int

Default

“str(60 * 60 * 24)”

Required

No

Max number of seconds you are allowed to be logged in with OAuth2. When theuser has been logged in >= this number, the user is automatically logged out.

Parser

int

Default

“4”

Required

No

This is the number of versions to display if a particular product has no ‘featured versions’. Then we use the active versions, but capped up to this number.

VERSIONS_WINDOW_DAYS
Parser

int

Default

“60”

Required

No

Number of days to look at for versions in crash reports. This is set for two months. If we haven’t gotten a crash report for some version in two months, then seems like that version isn’t active.

VERSIONS_COUNT_THRESHOLD
Parser

int

Default

“50”

Required

No

Minimum number of crash reports in the VERSIONS_WINDOW_DAYS to be considered as a valid version.

LESS_BINARY
Parser

str

Default

“path(“node_modules/.bin/lessc”)”

Required

No

UGLIFYJS_BINARY
Parser

str

Default

“path(“node_modules/.bin/uglifyjs”)”

Required

No

CSSMIN_BINARY
Parser

str

Default

“path(“node_modules/.bin/cssmin”)”

Required

No

CACHE_LOCATION
Parser

str

Default

“127.0.0.1:11211”

Required

No

CACHE_TIMEOUT
Parser

int

Default

“500”

Required

No

CACHE_KEY_PREFIX
Parser

str

Default

“socorro”

Required

No