docker-compose deployment of v0.6-branch

[masonlr]

Expected Behavior

Deploy feast v0.6-branch using docker-compose without errors (following https://docs.feast.dev/v/v0.6-branch/getting-started/deploying-feast/docker-compose).

Current Behavior

Errors occur when running the cells of feast/examples/basic/basic.ipynb

Steps to reproduce

Create GCP project, service account, bucket.

PROJECT=feast-example-project
SERVICE_ACCOUNT=feast-service-account
BUCKET=feast-bucket-31072020

gcloud projects create $PROJECT
gcloud config set project $PROJECT
gcloud iam service-accounts create $SERVICE_ACCOUNT

gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SERVICE_ACCOUNT@$PROJECT.iam.gserviceaccount.com --role roles/editor

gsutil mb gs://$BUCKET

cd infra/docker-compose/gcp-service-accounts
gcloud iam service-accounts keys create credentials.json --iam-account $SERVICE_ACCOUNT@$PROJECT.iam.gserviceaccount.com

Create .env file with:

# General
COMPOSE_PROJECT_NAME=feast
FEAST_VERSION=0.6.1
FEAST_REPOSITORY_VERSION=v0.6-branch

# Feast Core
FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core
FEAST_CORE_CONFIG=core.yml
FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=credentials.json

# Feast Serving
FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving
# Feast Serving - Batch (BigQuery)
FEAST_BATCH_SERVING_CONFIG=batch-serving.yml
FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=credentials.json
# Feast Serving - Online (Redis)
FEAST_ONLINE_SERVING_CONFIG=online-serving.yml

# Jupyter
FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=credentials.json

Update batch-serving.yaml to:

feast:
  core-host: core
  active-store: historical
  stores:
    - name: historical
      type: BIGQUERY
      # Changes required for batch serving to work
      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options
      config:
        project_id: feast-example-project
        dataset_id: dataset
        staging_location: gs://feast-bucket-31072020/test
        initial_retry_delay_seconds: 1
        total_timeout_seconds: 21600
      subscriptions:
        - name: "*"
          project: "*"
  job_store:
    redis_host: redis
    redis_port: 6379
  
grpc:
  port: 6567

Bring containers up using:

docker container prune -f && docker volume prune -f && docker network prune -f
docker pull gcr.io/kf-feast/feast-serving:0.6.1
docker pull gcr.io/kf-feast/feast-core:0.6.1
docker-compose -f docker-compose.yml -f docker-compose.online.yml -f docker-compose.batch.yml up

A java.lang.IllegalArgumentException error is shown in the core logs upon running client.apply(customer_fs) in the notebook:

$ docker logs feast_core_1
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

███████╗███████╗ █████╗ ███████╗████████╗
██╔════╝██╔════╝██╔══██╗██╔════╝╚══██╔══╝
█████╗  █████╗  ███████║███████╗   ██║   
██╔══╝  ██╔══╝  ██╔══██║╚════██║   ██║   
██║     ███████╗██║  ██║███████║   ██║   
╚═╝     ╚══════╝╚═╝  ╚═╝╚══════╝   ╚═╝   
                                         
 ██████╗ ██████╗ ██████╗ ███████╗        
██╔════╝██╔═══██╗██╔══██╗██╔════╝        
██║     ██║   ██║██████╔╝█████╗          
██║     ██║   ██║██╔══██╗██╔══╝          
╚██████╗╚██████╔╝██║  ██║███████╗        
 ╚═════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝        

2020-07-31 14:06:35.558  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Starting CoreApplication on 12f08993e2f0 with PID 1 (/opt/feast/feast-core.jar started by root in /)
2020-07-31 14:06:35.579 DEBUG 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Running with Spring Boot v2.0.9.RELEASE, Spring v5.0.13.RELEASE
2020-07-31 14:06:35.583  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : No active profile set, falling back to default profiles: default
2020-07-31 14:06:35.926  INFO 12f08993e2f0 --- [           main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7354b8c5: startup date [Fri Jul 31 14:06:35 UTC 2020]; root of context hierarchy
2020-07-31 14:06:40.170  INFO 12f08993e2f0 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2020-07-31 14:06:40.657  INFO 12f08993e2f0 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.kafka.annotation.KafkaBootstrapConfiguration' of type [org.springframework.kafka.annotation.KafkaBootstrapConfiguration$$EnhancerBySpringCGLIB$$a4fe9a98] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-07-31 14:06:40.764  INFO 12f08993e2f0 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$d14c8915] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-07-31 14:06:41.667  INFO 12f08993e2f0 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2020-07-31 14:06:41.708  INFO 12f08993e2f0 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
2020-07-31 14:06:41.749  INFO 12f08993e2f0 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2020-07-31 14:06:41.756  INFO 12f08993e2f0 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet Engine: Apache Tomcat/8.5.39
2020-07-31 14:06:41.778  INFO 12f08993e2f0 --- [ost-startStop-1] o.a.c.c.AprLifecycleListener             : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
2020-07-31 14:06:41.928  INFO 12f08993e2f0 --- [ost-startStop-1] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-31 14:06:41.928  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 6015 ms
2020-07-31 14:06:43.014  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.
2020-07-31 14:06:43.015  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration retrieval from the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.getConfiguration() instead.
2020-07-31 14:06:43.016  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration retrieval from the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.getConfiguration() instead.
2020-07-31 14:06:43.016  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.
2020-07-31 14:06:43.045  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.Flyway                             : Direct configuration of the Flyway object has been deprecated and will be removed in Flyway 6.0. Use Flyway.configure() instead.
2020-07-31 14:06:43.076  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.l.VersionPrinter                 : Flyway Community Edition 5.2.4 by Boxfuse
2020-07-31 14:06:43.107  INFO 12f08993e2f0 --- [ost-startStop-1] c.z.h.HikariDataSource                   : HikariPool-1 - Starting...
2020-07-31 14:06:43.311  INFO 12f08993e2f0 --- [ost-startStop-1] c.z.h.HikariDataSource                   : HikariPool-1 - Start completed.
2020-07-31 14:06:43.327  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.d.DatabaseFactory                : Database: jdbc:postgresql://db:5432/postgres (PostgreSQL 12.3)
2020-07-31 14:06:43.341  WARN 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.d.b.Database                     : Flyway upgrade recommended: PostgreSQL 12.3 is newer than this version of Flyway and support has not been tested.
2020-07-31 14:06:43.513  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbValidate                     : Successfully validated 7 migrations (execution time 00:00.080s)
2020-07-31 14:06:43.529  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.s.JdbcTableSchemaHistory         : Creating Schema History table: "public"."flyway_schema_history"
2020-07-31 14:06:43.563  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Current version of schema "public": << Empty Schema >>
2020-07-31 14:06:43.566  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 1 - Baseline
2020-07-31 14:06:43.643  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2 - RELEASE 0.6 Generalizing Source AND Extending FeatureSetJobStatus AND Feature Statistics
2020-07-31 14:06:43.690  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2.1 - Many Stores Per Job
2020-07-31 14:06:43.713  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2.2 - Subscription Migration
2020-07-31 14:06:43.724  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2.3 - Fix Primary Keys
2020-07-31 14:06:43.740  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2.4 - Store proto
2020-07-31 14:06:43.751  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Migrating schema "public" to version 2.5 - Fix Subscription MIgration
2020-07-31 14:06:43.764  INFO 12f08993e2f0 --- [ost-startStop-1] o.f.c.i.c.DbMigrate                      : Successfully applied 7 migrations to schema "public" (execution time 00:00.235s)
2020-07-31 14:06:44.314  INFO 12f08993e2f0 --- [ost-startStop-1] j.LocalContainerEntityManagerFactoryBean : Building JPA container EntityManagerFactory for persistence unit 'default'
2020-07-31 14:06:44.349  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.j.i.u.LogHelper                      : HHH000204: Processing PersistenceUnitInfo [
	name: default
	...]
2020-07-31 14:06:44.508  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.Version                              : HHH000412: Hibernate Core {5.3.6.Final}
2020-07-31 14:06:44.512  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.c.Environment                        : HHH000206: hibernate.properties not found
2020-07-31 14:06:44.890  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.a.c.Version                          : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2020-07-31 14:06:45.711  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.d.Dialect                            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL95Dialect
2020-07-31 14:06:45.951  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.e.j.e.i.LobCreatorBuilderImpl        : HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
2020-07-31 14:06:45.967  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.t.BasicTypeRegistry                  : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@573f34e5
2020-07-31 14:06:48.313  INFO 12f08993e2f0 --- [ost-startStop-1] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-07-31 14:06:49.694  INFO 12f08993e2f0 --- [ost-startStop-1] o.h.h.i.QueryTranslatorFactoryInitiator  : HHH000397: Using ASTQueryTranslatorFactory
2020-07-31 14:06:51.060  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'characterEncodingFilter' to: [/*]
2020-07-31 14:06:51.060  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'webMvcMetricsFilter' to: [/*]
2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2020-07-31 14:06:51.061  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'requestContextFilter' to: [/*]
2020-07-31 14:06:51.062  INFO 12f08993e2f0 --- [ost-startStop-1] .s.DelegatingFilterProxyRegistrationBean : Mapping filter: 'springSecurityFilterChain' to: [/*]
2020-07-31 14:06:51.062  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.FilterRegistrationBean         : Mapping filter: 'httpTraceFilter' to: [/*]
2020-07-31 14:06:51.063  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.ServletRegistrationBean        : Servlet metricsServlet mapped to [/metrics]
2020-07-31 14:06:51.068  INFO 12f08993e2f0 --- [ost-startStop-1] o.s.b.w.s.ServletRegistrationBean        : Servlet dispatcherServlet mapped to [/]
2020-07-31 14:06:51.712  INFO 12f08993e2f0 --- [           main] .s.s.UserDetailsServiceAutoConfiguration : 

Using generated security password: e2917407-3725-45d4-9e7a-e1772dcc4092

2020-07-31 14:06:51.993  INFO 12f08993e2f0 --- [           main] o.s.s.w.DefaultSecurityFilterChain       : Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@a565cbd, org.springframework.security.web.context.SecurityContextPersistenceFilter@481e91b6, org.springframework.security.web.header.HeaderWriterFilter@1daf3b44, org.springframework.security.web.authentication.logout.LogoutFilter@6f139fc9, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@3cbf1ba4, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@383864d5, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@6fc29daa, org.springframework.security.web.session.SessionManagementFilter@8a2a6a, org.springframework.security.web.access.ExceptionTranslationFilter@3ee69ad8, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@3cb8c8ce]
2020-07-31 14:06:52.253  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-31 14:06:52.531  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@7354b8c5: startup date [Fri Jul 31 14:06:35 UTC 2020]; root of context hierarchy
2020-07-31 14:06:52.616  WARN 12f08993e2f0 --- [           main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2020-07-31 14:06:52.680  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2020-07-31 14:06:52.681  INFO 12f08993e2f0 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2020-07-31 14:06:52.737  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-31 14:06:52.737  INFO 12f08993e2f0 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-31 14:06:54.006  INFO 12f08993e2f0 --- [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-07-31 14:06:54.039  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/health],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2020-07-31 14:06:54.040  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator/info],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(javax.servlet.http.HttpServletRequest,java.util.Map<java.lang.String, java.lang.String>)
2020-07-31 14:06:54.042  INFO 12f08993e2f0 --- [           main] s.b.a.e.w.s.WebMvcEndpointHandlerMapping : Mapped "{[/actuator],methods=[GET],produces=[application/vnd.spring-boot.actuator.v2+json || application/json]}" onto protected java.util.Map<java.lang.String, java.util.Map<java.lang.String, org.springframework.boot.actuate.endpoint.web.Link>> org.springframework.boot.actuate.endpoint.web.servlet.WebMvcEndpointHandlerMapping.links(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2020-07-31 14:06:54.239  INFO 12f08993e2f0 --- [           main] o.a.k.c.a.AdminClientConfig              : AdminClientConfig values: 
	bootstrap.servers = [kafka:9092, localhost:9094]
	client.dns.lookup = default
	client.id = 
	connections.max.idle.ms = 300000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 15000
	retries = 5
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS

2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka version: 2.3.0
2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka commitId: fc1aaa116b661c8a
2020-07-31 14:06:54.344  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka startTimeMs: 1596204414340
2020-07-31 14:06:54.892 ERROR 12f08993e2f0 --- [           main] o.s.k.c.KafkaAdmin                       : Failed to create topics
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
2020-07-31 14:06:54.903  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2020-07-31 14:06:54.908  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'dataSource' has been autodetected for JMX exposure
2020-07-31 14:06:54.932  INFO 12f08993e2f0 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2020-07-31 14:06:55.019  INFO 12f08993e2f0 --- [           main] o.s.c.s.DefaultLifecycleProcessor        : Starting beans in phase 2147483547
2020-07-31 14:06:55.049  INFO 12f08993e2f0 --- [           main] o.a.k.c.c.ConsumerConfig                 : ConsumerConfig values: 
	allow.auto.create.topics = true
	auto.commit.interval.ms = 5000
	auto.offset.reset = latest
	bootstrap.servers = [kafka:9092, localhost:9094]
	check.crcs = true
	client.dns.lookup = default
	client.id = 
	client.rack = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = true
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = core-service-null
	group.instance.id = null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class feast.core.util.KafkaSerialization$ProtoDeserializer

2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka version: 2.3.0
2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka commitId: fc1aaa116b661c8a
2020-07-31 14:06:55.119  INFO 12f08993e2f0 --- [           main] o.a.k.c.u.AppInfoParser                  : Kafka startTimeMs: 1596204415119
2020-07-31 14:06:55.124  INFO 12f08993e2f0 --- [           main] o.a.k.c.c.KafkaConsumer                  : [Consumer clientId=consumer-1, groupId=core-service-null] Subscribed to topic(s): feast-specs-ack
2020-07-31 14:06:55.130  INFO 12f08993e2f0 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService
2020-07-31 14:06:55.139  INFO 12f08993e2f0 --- [           main] o.s.c.s.DefaultLifecycleProcessor        : Starting beans in phase 2147483647
2020-07-31 14:06:55.151  WARN 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.NetworkClient                    : [Consumer clientId=consumer-1, groupId=core-service-null] Connection to node -2 (localhost/127.0.0.1:9094) could not be established. Broker may not be available.
2020-07-31 14:06:55.175  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.Metadata                         : [Consumer clientId=consumer-1, groupId=core-service-null] Cluster ID: Rlqi6O23SFai-FI3_al8Sg
2020-07-31 14:06:55.195  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.AbstractGrpcServerFactory    : Registered gRPC service: feast.core.CoreService, bean: coreServiceImpl, class: feast.core.grpc.CoreServiceImpl
2020-07-31 14:06:55.202  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.AbstractGrpcServerFactory    : Registered gRPC service: grpc.health.v1.Health, bean: healthServiceImpl, class: feast.core.grpc.HealthServiceImpl
2020-07-31 14:06:55.673  INFO 12f08993e2f0 --- [           main] n.d.b.g.s.s.GrpcServerLifecycle          : gRPC Server started, listening on address: *, port: 6565
2020-07-31 14:06:55.713  INFO 12f08993e2f0 --- [           main] s.a.ScheduledAnnotationBeanPostProcessor : No TaskScheduler/ScheduledExecutorService bean found for scheduled processing
2020-07-31 14:06:55.749  INFO 12f08993e2f0 --- [           main] o.a.c.h.Http11NioProtocol                : Starting ProtocolHandler ["http-nio-8080"]
2020-07-31 14:06:55.751  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Polling for new jobs...
2020-07-31 14:06:55.762  INFO 12f08993e2f0 --- [           main] o.a.t.u.n.NioSelectorPool                : Using a shared selector for servlet write/read
2020-07-31 14:06:55.925  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : No jobs found.
2020-07-31 14:06:55.964  INFO 12f08993e2f0 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''
2020-07-31 14:06:55.969  INFO 12f08993e2f0 --- [           main] f.c.CoreApplication                      : Started CoreApplication in 21.761 seconds (JVM running for 24.3)
2020-07-31 14:06:56.246  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Discovered group coordinator kafka:9092 (id: 2147482646 rack: null)
2020-07-31 14:06:56.222  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Revoking previously assigned partitions []
2020-07-31 14:06:56.222  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: []
2020-07-31 14:06:56.223  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] (Re-)joining group
2020-07-31 14:06:56.263  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] (Re-)joining group
2020-07-31 14:06:59.390  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.AbstractCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Successfully joined group with generation 1
2020-07-31 14:06:59.399  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Setting newly assigned partitions: feast-specs-ack-0
2020-07-31 14:06:59.424  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.ConsumerCoordinator          : [Consumer clientId=consumer-1, groupId=core-service-null] Found no committed offset for partition feast-specs-ack-0
2020-07-31 14:06:59.456  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.a.k.c.c.i.SubscriptionState            : [Consumer clientId=consumer-1, groupId=core-service-null] Resetting offset for partition feast-specs-ack-0 to offset 0.
2020-07-31 14:06:59.461  INFO 12f08993e2f0 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [feast-specs-ack-0]
2020-07-31 14:07:25.894  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Polling for new jobs...
2020-07-31 14:07:25.922  INFO 12f08993e2f0 --- [pool-3-thread-1] f.c.s.JobCoordinatorService              : Creating/Updating 2 jobs...
2020-07-31 14:07:25.926 AUDIT 12f08993e2f0 --- [pool-4-thread-1] f.c.l.AuditLogger                        : {action=SUBMIT, detail=Building graph and submitting to DirectRunner, id=kafka-2111741832-to-historical-1596204445910, resource=JOB, timestamp=Fri Jul 31 14:07:25 UTC 2020}
2020-07-31 14:07:25.926 AUDIT 12f08993e2f0 --- [pool-4-thread-2] f.c.l.AuditLogger                        : {action=SUBMIT, detail=Building graph and submitting to DirectRunner, id=kafka-2111741832-to-online-1596204445919, resource=JOB, timestamp=Fri Jul 31 14:07:25 UTC 2020}
2020-07-31 14:07:26.866  INFO 12f08993e2f0 --- [pool-4-thread-2] f.i.ImportJob                            : Starting import job with settings: 
Current Settings:
  appName: DirectRunnerJobManager
  blockOnRun: false
  defaultFeastProject: default
  enforceEncodability: true
  enforceImmutability: true
  gcsPerformanceMetrics: false
  jobName: kafka-2111741832-to-online-1596204445919
  optionsId: 1
  project: 
  runner: class org.apache.beam.runners.direct.DirectRunner
  sourceJson: {
  "type": "KAFKA",
  "kafkaSourceConfig": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-features"
  }
}
  specsStreamingUpdateConfigJson: {
  "source": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-specs"
  },
  "ack": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-specs-ack"
  }
}
  stableUniqueNames: WARNING
  storesJson: [{
  "name": "online",
  "type": "REDIS",
  "subscriptions": [{
    "name": "*",
    "project": "*"
  }],
  "redisConfig": {
    "host": "redis",
    "port": 6379
  }
}]

2020-07-31 14:07:26.866  INFO 12f08993e2f0 --- [pool-4-thread-1] f.i.ImportJob                            : Starting import job with settings: 
Current Settings:
  appName: DirectRunnerJobManager
  blockOnRun: false
  defaultFeastProject: default
  enforceEncodability: true
  enforceImmutability: true
  gcsPerformanceMetrics: false
  jobName: kafka-2111741832-to-historical-1596204445910
  optionsId: 0
  project: 
  runner: class org.apache.beam.runners.direct.DirectRunner
  sourceJson: {
  "type": "KAFKA",
  "kafkaSourceConfig": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-features"
  }
}
  specsStreamingUpdateConfigJson: {
  "source": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-specs"
  },
  "ack": {
    "bootstrapServers": "kafka:9092,localhost:9094",
    "topic": "feast-specs-ack"
  }
}
  stableUniqueNames: WARNING
  storesJson: [{
  "name": "historical",
  "type": "BIGQUERY",
  "subscriptions": [{
    "name": "*",
    "project": "*"
  }],
  "bigqueryConfig": {
    "projectId": "feast-example-project",
    "datasetId": "dataset",
    "stagingLocation": "gs://feast-bucket-31072020/test",
    "initialRetryDelaySeconds": 1,
    "totalTimeoutSeconds": 21600
  }
}]

2020-07-31 14:07:27.685 ERROR 12f08993e2f0 --- [pool-4-thread-1] f.c.j.d.DirectRunnerJobManager           : Error submitting job
java.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 <= offset < size
	at org.apache.beam.sdk.transforms.windowing.FixedWindows.<init>(FixedWindows.java:65)
	at org.apache.beam.sdk.transforms.windowing.FixedWindows.of(FixedWindows.java:50)
	at feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:135)
	at feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:49)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
	at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
	at feast.ingestion.ImportJob.runPipeline(ImportJob.java:167)
	at feast.core.job.direct.DirectRunnerJobManager.runPipeline(DirectRunnerJobManager.java:164)
	at feast.core.job.direct.DirectRunnerJobManager.startJob(DirectRunnerJobManager.java:83)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:48)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:29)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2020-07-31 14:07:27.686 ERROR 12f08993e2f0 --- [pool-4-thread-1] f.c.j.CreateJobTask                      : Error running ingestion job: java.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 <= offset < size
2020-07-31 14:07:27.686 AUDIT 12f08993e2f0 --- [pool-4-thread-1] f.c.l.AuditLogger                        : {action=STATUS_CHANGE, detail=Job failed to be submitted to runner DirectRunner. Job status changed to ERROR., id=kafka-2111741832-to-historical-1596204445910, resource=JOB, timestamp=Fri Jul 31 14:07:27 UTC 2020}
2020-07-31 14:07:27.939  INFO 12f08993e2f0 --- [pool-4-thread-2] i.l.c.EpollProvider                      : Starting without optional epoll library
2020-07-31 14:07:27.943  INFO 12f08993e2f0 --- [pool-4-thread-2] i.l.c.KqueueProvider                     : Starting without optional kqueue library
2020-07-31 14:07:28.971  INFO 12f08993e2f0 --- [pool-4-thread-2] o.a.k.c.c.ConsumerConfig                 : ConsumerConfig values: 

Specifications

docker desktop v2.2.0.4 on macOS 10.15.5

Possible Solution

Include write_triggering_frequency_seconds in batch-serving.yaml:

feast:
  core-host: core
  active-store: historical
  stores:
    - name: historical
      type: BIGQUERY
      # Changes required for batch serving to work
      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options
      config:
        project_id: feast-example-project
        dataset_id: dataset
        staging_location: gs://feast-bucket-31072020/test
        initial_retry_delay_seconds: 1
        total_timeout_seconds: 21600
        write_triggering_frequency_seconds: 1
      subscriptions:
        - name: "*"
          project: "*"
  job_store:
    redis_host: redis
    redis_port: 6379
  
grpc:
  port: 6567

Adding the trigger frequency setting changes the error to:

$ docker logs feast_core_1

...

2020-07-31 14:15:13.157 ERROR d70f6bada002 --- [pool-4-thread-2] f.c.j.d.DirectRunnerJobManager           : Error submitting job
java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
	at org.apache.beam.sdk.schemas.utils.ReflectUtils.getMethods(ReflectUtils.java:77)
	at org.apache.beam.sdk.schemas.AutoValueSchema$AbstractGetterTypeSupplier.get(AutoValueSchema.java:46)
	at org.apache.beam.sdk.schemas.utils.StaticSchemaInference.schemaFromClass(StaticSchemaInference.java:83)
	at org.apache.beam.sdk.schemas.utils.JavaBeanUtils.schemaFromJavaBeanClass(JavaBeanUtils.java:63)
	at org.apache.beam.sdk.schemas.AutoValueSchema.schemaFor(AutoValueSchema.java:114)
	at org.apache.beam.sdk.schemas.annotations.DefaultSchema$DefaultSchemaProvider.schemaFor(DefaultSchema.java:131)
	at org.apache.beam.sdk.schemas.SchemaRegistry.lambda$getSchema$0(SchemaRegistry.java:256)
	at org.apache.beam.sdk.schemas.SchemaRegistry.getProviderResult(SchemaRegistry.java:239)
	at org.apache.beam.sdk.schemas.SchemaRegistry.getSchema(SchemaRegistry.java:256)
	at org.apache.beam.sdk.values.PCollection.inferCoderOrFail(PCollection.java:159)
	at org.apache.beam.sdk.values.PCollection.finishSpecifyingOutput(PCollection.java:94)
	at org.apache.beam.sdk.runners.TransformHierarchy.setOutput(TransformHierarchy.java:210)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:540)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
	at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
	at feast.ingestion.transform.ReadFromSource.expand(ReadFromSource.java:91)
	at feast.ingestion.transform.ReadFromSource.expand(ReadFromSource.java:37)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
	at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
	at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:184)
	at feast.ingestion.ImportJob.runPipeline(ImportJob.java:117)
	at feast.core.job.direct.DirectRunnerJobManager.runPipeline(DirectRunnerJobManager.java:164)
	at feast.core.job.direct.DirectRunnerJobManager.startJob(DirectRunnerJobManager.java:83)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:48)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:29)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2020-07-31 14:15:13.157 ERROR d70f6bada002 --- [pool-4-thread-2] f.c.j.CreateJobTask                      : Error running ingestion job: java.util.ConcurrentModificationException

[masonlr]

When using

# General
COMPOSE_PROJECT_NAME=feast
FEAST_VERSION=latest
FEAST_REPOSITORY_VERSION=v0.6-branch

# Feast Core
FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core
FEAST_CORE_CONFIG=core.yml
FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=credentials.json

# Feast Serving
FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving
# Feast Serving - Batch (BigQuery)
FEAST_BATCH_SERVING_CONFIG=batch-serving.yml
FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=credentials.json
# Feast Serving - Online (Redis)
FEAST_ONLINE_SERVING_CONFIG=online-serving.yml

# Jupyter
FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=credentials.json

with

feast:
  core-host: core
  active-store: historical
  stores:
    - name: historical
      type: BIGQUERY
      # Changes required for batch serving to work
      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options
      config:
        project_id: feast-example-project
        dataset_id: dataset
        staging_location: gs://feast-bucket-31072020/test
        initial_retry_delay_seconds: 1
        total_timeout_seconds: 21600
        write_triggering_frequency_seconds: 1
      subscriptions:
        - name: "*"
          project: "*"
  job_store:
    redis_host: redis
    redis_port: 6379
  
grpc:
  port: 6567

the feature set hangs with STATUS_PENDING and a kafka connection error is showing in the logs:

$ docker logs feast_core_1

...

2020-07-31 14:25:58.753 ERROR 75dd5d8cdca0 --- [           main] o.s.k.c.KafkaAdmin                       : Failed to create topics
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

...

[masonlr]

When using

# General
COMPOSE_PROJECT_NAME=feast
FEAST_VERSION=latest
FEAST_REPOSITORY_VERSION=v0.6-branch

# Feast Core
FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core
FEAST_CORE_CONFIG=core.yml
FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=credentials.json

# Feast Serving
FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving
# Feast Serving - Batch (BigQuery)
FEAST_BATCH_SERVING_CONFIG=batch-serving.yml
FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=credentials.json
# Feast Serving - Online (Redis)
FEAST_ONLINE_SERVING_CONFIG=online-serving.yml

# Jupyter
FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=credentials.json

feast:
  core-host: core
  active-store: historical
  stores:
    - name: historical
      type: BIGQUERY
      # Changes required for batch serving to work
      # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options
      config:
        project_id: feast-example-project
        dataset_id: dataset
        staging_location: gs://feast-bucket-31072020/test
        initial_retry_delay_seconds: 1
        total_timeout_seconds: 21600
      subscriptions:
        - name: "*"
          project: "*"
  job_store:
    redis_host: redis
    redis_port: 6379
  
grpc:
  port: 6567

I'm getting similar errors

$ docker logs feast_core_1
...
2020-07-31 14:32:07.001 ERROR ef73aa55857f --- [           main] o.s.k.c.KafkaAdmin                       : Failed to create topics
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

...

2020-07-31 14:32:39.992 ERROR ef73aa55857f --- [pool-4-thread-1] f.c.j.d.DirectRunnerJobManager           : Error submitting job
java.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 <= offset < size
	at org.apache.beam.sdk.transforms.windowing.FixedWindows.<init>(FixedWindows.java:65)
	at org.apache.beam.sdk.transforms.windowing.FixedWindows.of(FixedWindows.java:50)
	at feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:135)
	at feast.storage.connectors.bigquery.writer.BigQueryWrite.expand(BigQueryWrite.java:49)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490)
	at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:368)
	at feast.ingestion.ImportJob.runPipeline(ImportJob.java:167)
	at feast.core.job.direct.DirectRunnerJobManager.runPipeline(DirectRunnerJobManager.java:164)
	at feast.core.job.direct.DirectRunnerJobManager.startJob(DirectRunnerJobManager.java:83)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:48)
	at feast.core.job.CreateJobTask.call(CreateJobTask.java:29)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
2020-07-31 14:32:39.992 ERROR ef73aa55857f --- [pool-4-thread-1] f.c.j.CreateJobTask                      : Error running ingestion job: java.lang.IllegalArgumentException: FixedWindows WindowingStrategies must have 0 <= offset < size
2020-07-31 14:32:39.993 AUDIT ef73aa55857f --- [pool-4-thread-1] f.c.l.AuditLogger                        : {action=STATUS_CHANGE, detail=Job failed to be submitted to runner DirectRunner. Job status changed to ERROR., id=kafka-2111741832-to-historical-1596205958215, resource=JOB, timestamp=Fri Jul 31 14:32:39 UTC 2020}
2020-07-31 14:32:40.258  INFO ef73aa55857f --- [pool-4-thread-2] i.l.c.EpollProvider                      : Starting without optional epoll library
2020-07-31 14:32:40.261  INFO ef73aa55857f --- [pool-4-thread-2] i.l.c.KqueueProvider                     : Starting without optional kqueue library

...

the feature set gives STATUS_READY but after the ingestion cell is run, the df = job.to_dataframe() retrieval cell gives

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-33-6b4ac8e41ab2> in <module>
----> 1 df = job.to_dataframe()

/feast/sdk/python/feast/job.py in to_dataframe(self, timeout_sec)
    149                 Pandas DataFrame of the feature values.
    150         """
--> 151         records = [r for r in self.result(timeout_sec=timeout_sec)]
    152         return pd.DataFrame.from_records(records)
    153 

/feast/sdk/python/feast/job.py in <listcomp>(.0)
    149                 Pandas DataFrame of the feature values.
    150         """
--> 151         records = [r for r in self.result(timeout_sec=timeout_sec)]
    152         return pd.DataFrame.from_records(records)
    153 

/feast/sdk/python/feast/job.py in result(self, timeout_sec)
    120             Iterable of Avro rows.
    121         """
--> 122         uris = self.get_avro_files(timeout_sec)
    123         for file_uri in uris:
    124             file_obj = get_staging_client(file_uri.scheme).download_file(file_uri)

/feast/sdk/python/feast/job.py in get_avro_files(self, timeout_sec)
     97 
     98         if self.job_proto.error:
---> 99             raise Exception(self.job_proto.error)
    100 
    101         if self.job_proto.data_format != DATA_FORMAT_AVRO:

Exception: Unable to load entity table to BigQuery: io.grpc.StatusRuntimeException: INTERNAL: Failed to load entity dataset into store: com.google.cloud.bigquery.BigQueryException: Not found: Dataset feast-example-project:dataset

[pyalex]

write_triggering_frequency_seconds is missing in helm & docker compose configs.
Fixed by #917
Also more informative exception would be thrown in case write_triggering_frequency_seconds would still be missing.

[pyalex]

Also please create dataset manually, it's not being created programmatically right now. @masonlr