4
$\begingroup$

I am not entirely sure if this is on-topic here, so please let me know if it is not. I keep seeing the idea of YAML files pop up while reading machine learning literature. My question is, what exactly is a YAML file, and how does it relate to machine learning and data science projects?

||improve this question
$\endgroup$
4
$\begingroup$

There are numerous ways to serialise data. For instance, JSON, XML, and CSV are possible approaches for serialisation. Another way is YAML. It is human-readable which means it contains readable text which is written using usual words. This stands against binary serialisation. Whenever you open the file, you will not be able to see what is written, because it is serialised in binary format and not as a human-readable format. As you may find it here, it is used for configuration files mostly. In ML context depending on your task, you may face different file format. For instance, one of the widely used formats for supervised ML tasks is CSV though you may find others use their favorite encodings. You may also want to make your own serialisation too which is possible among in-company operations is rare.

||improve this answer
$\endgroup$
  • $\begingroup$ Just a quick follow up - what is serialisation? $\endgroup$ – user67797 Jul 6 '19 at 3:02
  • $\begingroup$ @QFII I don't know whether you've worked with JSON or XML but they are a kind of way to store your objects that reside in RAM to be stored on a disk. RAM is volatile while disks are stable and the contents stored on them can be retrieved. $\endgroup$ – Media Jul 6 '19 at 3:38
1
$\begingroup$

In addition to data a yaml file is often used as a human readable configuration file ... here is an example of a docker compose yaml file

# version: "3.5"
version: '3'
# version: '2'
services:


  nodejs-admin:
    image: ${GKE_APP_IMAGE_ADMIN}
	    hostname: admin
	    container_name: loud_admin
	    restart: always
	    depends_on:
	      - loudmongo
	      # - loudmail
	      - loudmail
	    volumes:
	      - /cryptdata4/var/log/loudlog-admin:/loudlog-admin
	      - /cryptdata5/var/log/blobs:/blobs
	      - /cryptdata5/var:/cryptdata5/var
	      - /cryptdata5/var/tools:/tools
	      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
	      # - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
	      - ${TMPDIR_GRAND_PARENT}/${bundleNormal}/loud-build/${PROJECT_ID}/webapp/admin/bundle:/tmp
      # - $SOURCE_REPO_DIR/tests:/tmp/tests
	    environment:
	      - MONGO_SERVICE_HOST=loudmongo
	      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT
      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/admin
	      - METEOR_SETTINGS=${METEOR_SETTINGS}
      # when sending to port 587 will NOT get routed through email content filter however port 25 will get routed
      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:ignore_this@loudmail:587/
	      # - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:ignore_this@loudmail:25/
      - GKE_NOTIF_TASK_OVERDUE=$GKE_NOTIF_TASK_OVERDUE
	      - GKE_NOTIF_PLANS_RECUR=$GKE_NOTIF_PLANS_RECUR
      - GKE_NOTIF_SHIFTS_RUN=$GKE_NOTIF_SHIFTS_RUN
	      - GKE_NOTIF_LOAD_RUN=$GKE_NOTIF_LOAD_RUN
      - GKE_NOTIF_ORGS_HEALTH_RUN=$GKE_NOTIF_ORGS_HEALTH_RUN
	    links:
	      - loudmongo
	      # - loudmail
	      - loudmail
	    ports:
	     - 127.0.0.1:${PORT_ADMIN}:3001
#   networks:
#    - loudthink-network
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

  nodejs-enduser:
    # image: ${GKE_APP_REPO_PREFIX}/${PROJECT_ID}/loudweb-enduser
    image: ${GKE_APP_IMAGE_ENDUSER}
	    hostname: enduser
	    container_name: loud_enduser
	    restart: always
	    depends_on:
	      - nodejs-admin
	      - loudmongo
	      # - loudmail
	      - loudmail
	    volumes:
	      - /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
	      - /cryptdata5/var/log/blobs:/blobs
	      # - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
	      # - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
	      # - ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
	      - ${TMPDIR_GRAND_PARENT}/${bundleNormal}/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
      - ${TMPDIR_GRAND_PARENT}/${bundleNormal}/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-cert.pem:/private/config/apn-cert.pem
	      - ${TMPDIR_GRAND_PARENT}/${bundleNormal}/loud-build/${PROJECT_ID}/webapp/admin/bundle/programs/server/assets/app/config/apn-key.pem:/private/config/apn-key.pem
    environment:
      - MONGO_SERVICE_HOST=loudmongo
      - MONGO_SERVICE_PORT=$GKE_MONGO_PORT
	      - MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/admin
      - METEOR_SETTINGS=${METEOR_SETTINGS}
	      # when sending to port 587 will NOT get routed through email content filter however port 25 will get routed
	      - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:ignore@loudmail:587/
      # - MAIL_URL=smtp://support@${GKE_DOMAIN_NAME}:ignore@loudmail:25/
	    links:
	      - loudmongo
	      # - loudmail
	      - loudmail
	    ports:
	     - 127.0.0.1:${PORT_ENDUSER}:3000
#   networks:
#    - loudthink-network
    working_dir: /tmp
    command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

  loudmongo:
    # image: mongo
    # image: mongo:3.6
    image: $GKE_MONGO_IMAGE
	    hostname: mongo
	    container_name: loud_mongo
	    restart: always
	    ports:
	     - 127.0.0.1:$GKE_MONGO_PORT:$GKE_MONGO_PORT
#   logpath:
#    - /data/logs/mongo.log
    volumes:
     - /cryptdata7/var/data/db:/data/db
     # - /cryptdata7/var/data/logs:/data/logs




  loud-devops:
    # image: dind
    image: localhost:5000/hygge/loudweb-dind/1904082202
    hostname: devops
    container_name: loud_devops
    restart: always
    ports:
      - 127.0.0.1:9000:9000
#     expose:
#      - 9000
#   networks:
#    - loudthink-network
#     environment:
#       - SSH_AUTH_SOCK=$SSH_AUTH_SOCK
	    volumes:
	      - /var/run/docker.sock:/var/run/docker.sock   
	      - /usr/bin/docker:/usr/bin/docker   
	      # - $SSH_AUTH_SOCK:/ssh-agent 
      - /usr/lib/x86_64-linux-gnu/libltdl.so.7:/usr/lib/x86_64-linux-gnu/libltdl.so.7  
      - /usr/lib/x86_64-linux-gnu/libgpm.so.2:/usr/lib/x86_64-linux-gnu/libgpm.so.2
      - /home/khepri/src/github.com/loudthink:/inner_home/khepri/src/github.com/loudthink 
      - /home/khepri/.docker:/inner_home/khepri/.docker
      - /cryptdata5/var/tools/inner_home/khepri:/inner_home/khepri
      - $GKE_DIND_SUPERVISOR_LOG_DIR:/var/log/supervisor
	      - /cryptdata5/var/tools/usr/local/go:/usr/local/go
	      - /cryptdata6/var/log/tmp/khepri01:/cryptdata6/var/log/tmp/khepri01
	      - /cryptdata6/var/log/tmp/shared:/cryptdata6/var/log/tmp/shared
	      - /cryptdata5/var/tools/usr/local/bin:/usr/local/bin 
	      - /cryptdata:/cryptdata
	      - /cryptdata2:/cryptdata2
	      - /cryptdata4:/cryptdata4
	      - /cryptdata5:/cryptdata5
	      - /cryptdata6:/cryptdata6
	      - /cryptdata7:/cryptdata7
	      # - /etc/letsencrypt/live:/etc/letsencrypt/live
	      # - /etc/letsencrypt/archive/medssenger-dev.medstack.net:/etc/letsencrypt/archive/medssenger-dev.medstack.net
	      - /usr/local/ssl:/usr/local/ssl
	      #  following line HOME is only used during install as referenced in loudspeed/build/bin/setup_devops_sudo_cmds.sh to copy over .ssh files
	      # - ${HOME}:${HOME}
      # - /var/lib/docker:/var/lib/docker
    command: /usr/bin/supervisord -c /etc/supervisor/supervisord.conf



  loudmail:
    image: tvial/docker-mailserver:latest
    hostname: mail
    domainname: ${GKE_DOMAIN_NAME}
	    container_name: loud_mail
	    restart: always
	    environment:
	      - PERMIT_DOCKER=network
	      - SSL_TYPE=letsencrypt
	      - ONE_DIR=1
	      - DMS_DEBUG=1
	      - SPOOF_PROTECTION=0
	      - REPORT_RECIPIENT=1
	      - ENABLE_SPAMASSASSIN=0
	      - ENABLE_CLAMAV=0
	      - ENABLE_FAIL2BAN=1
	      - ENABLE_POSTGREY=0
	    cap_add:
	      - NET_ADMIN
	      - SYS_PTRACE
	    ports:
	    - "25:25"
	    - "587:587"
	    - "465:465"
	    volumes:
	      - ${GKE_MAIL_EHOOK_DIR}/data/:/var/mail/
      # - ${GKE_MAIL_EHOOK_DIR}/state/:/var/mail-state/
	      - ${GKE_MAIL_EHOOK_DIR}/state/:${GKE_MAIL_EHOOK_INNER_STATE_DIR}/
	      - ${GKE_MAIL_EHOOK_DIR}/config/:/tmp/docker-mailserver/
      - ${GKE_MAIL_EHOOK_OUTPUT_DIR}/:${GKE_MAIL_EHOOK_INNER_OUTPUT_DIR}/
      - ${GKE_MAIL_EHOOK_HOST_DIR}/:/local_ehook/
	      # - /etc/letsencrypt/:/etc/letsencrypt/
	      - ${GKE_LETSENCRYPT_DIR}/:/etc/letsencrypt/
      - ${GKE_MAIL_EHOOK_DIR}/log/:/var/log/mail/

As with any configuration text file the notion of template is useful ... for example if your yaml file called GKE_COMPOSE_YAML contains say 

foobar: some_token

you can update a hardcoded some_token by doing a string search and replace during your pre-process stage 

sed -i -- "s|some_token|${my_current_token}|g"   $GKE_COMPOSE_YAML

depending on what your process is using to parse your yaml file it may permit embedding environment variables into your yaml file as per following snippet from a yaml file 

    foobar: $my_current_token

when your env contains var my_current_token which may have been defined using

export my_current_token=something.cool.here
||improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy