问题

加入配置设置并输入Apachespark配置文本区新设置取代现有设置而不是附加

版本化

数据布列克斯运行时间5.1和5.1

因果

集群重开时,集群读取配置文件设置集群UI并覆盖默认设置

例举,当添加extraJavaOptions切入点spark配置文本区 :

spark.executor.extraJavaOptions -  javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus  _jmx_exporter/jmx_prometheus_javaagent.yml

后进sparkUI>环境>spark属性下方spark.executor.extraJavaOptions中新加配置设置显示

-javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus  _jmx_exporter/jmx_prometheus_javaagent.yml

任何现有设置都删除

关于引用,默认设置为:

-Djava.io.tmpdir=/local_disk0/tmp -XX:ReservedCodeCacheSize=256m -  XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-executor-1 -  Djava.security.properties=/databricks/spark/dbconf/java/extra.security -XX:+PrintFlagsFinal -  XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -  Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty  peFactoryImpl -  Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.Documen  tBuilderFactoryImpl -  Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFact  oryImpl -  Djavax.xml.validation.SchemaFactory=https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory-  Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -  Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMX  SImplementationSourceImpl

求解

添加新配置设置spark.executor.extraJavaOptions免损缺省设置 :

1. 内sparkUI>环境>spark属性选择并复制默认设置的所有属性spark.executor.extraJavaOptions.
2. 点击编辑.
3. 内spark配置文本区集群>集群名>高级选项>spark实战)粘贴默认设置
4. 附加默认设置下方的新配置设置
5. 点击文本区外,然后点击确认.
6. 重开集群

举个例子 假设你把下方设置粘入spark配置文本区新建配置设置附于默认设置

spark.executor.extraJavaOptions = -Djava.io.tmpdir=/local_disk0/tmp -  XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -Ddatabricks.serviceName=spark-  executor-1 -Djava.security.properties=/databricks/spark/dbconf/java/extra.security -  XX:+PrintFlagsFinal -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xss4m -  Djavax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.Dataty  peFactoryImpl -  Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentB  uilderFactoryImpl -  Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactor  yImpl -  Djavax.xml.validation.SchemaFactory:https://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory -  Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -  Dorg.w3c.dom.DOMImplementationSourceList=com.sun.org.apache.xerces.internal.dom.DOMXSImplem  entationSourceImpl -  javaagent:/opt/prometheus_jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/prometheus_jm  x_exporter/jmx_prometheus_javaagent.yml

重开集群后,默认设置和新加配置设置出现sparkUI>环境>spark属性.