We're on to Part 3. In Part 1, we deployed MicroBOSH that enables us to deploy BOSH. In Part 2, we deployed BOSH which got us used to using manifest files. The manifest rules all. This is all adapted from the official documentation located at Deploying Cloud Foundry on vSphere.
This tutorial will go over the steps it takes to deploy Cloud Foundry on vSphere. Here are the proper steps:
Part 1 - How to install MicroBOSH
Part 2 - Deploy BOSH with MicroBOSH
Part 3 - Deploy Cloud Foundry with BOSH
NOTE: This final part has not been 100% verified. The deployment of this Cloud Foundry manifest requires the following infrastructure: 72 vCPUs, 200GB of RAM, and 1 TB of storage. Of course, my home lab doesn't support this so I kept getting timeout errors during the deployment. My problem is that my vCenter server can't keep up with the deployment speed. My vCenter Server Virtual Appliance is only configured with 8GB of RAM but it needs something much beefier to be able to handle the amount of requests. I believe this is the case because my vSphere Web Client would lose connection for a few minutes and when the deployment stopped, the Web Client started responding again. If you do test out this manifest below, please confirm it by mentioning it in the comments below. I will hope to test this in a real lab environment soon.
1. If you didn't complete the last step from part 2, do that now. Set your BOSH target to your newly created director with bosh target 192.168.50.145 and view the status with bosh status
administrator@originalbosh:/bosh/deployments/bosh$ bosh target 192.168.50.145 Target set to `BOSH-director' Your username: admin Enter password: ***** Logged in as `admin'
administrator@originalbosh:/bosh/deployments/bosh$ bosh status
Config
/home/administrator/.bosh_config
Director
Name BOSH-director
URL https://192.168.50.145:25555
Version 1.2579.0 (release:4fef83a2 bosh:4fef83a2)
User admin
UUID c3119c14-ba8c-4293-8d6e-83f76d94cd46
CPI vsphere
dns disabled
compiled_package_cache disabled
snapshots disabled
Deployment
not set
2. The director needs a stemcell in order to deploy Cloud Foundry. Let's upload our ubuntu stemcell once again
bosh upload stemcell /bosh/stemcells/bosh-stemcell-2427-vsphere-esxi-ubuntu.tgz
3. Add a Cloud Foundry public release to the releases folders.
cd /bosh/releases git clone https://github.com/cloudfoundry/cf-release.git
4. Find the latest release. Change to the releases directory and look for appcloud-XXX.yml. At the time of this writing, appcloud-131.yml is the latest.
cd /bosh/releases/cf-release/releases ls
5. Upload the latest release
bosh upload release /bosh/releases/cf-release/releases/appcloud-131.yml
I'm not going to show the output because it's super long.
6. now it's time to create our manifest. Take a look at the sample manifest form from Cloud Foundry documentation.
Here is my adapted manifest, changing IPs, passwords, VM Versions, etc. All you need to change are the IP addresses and the passwords. You can copy/paste or download the manifest below the code.
---
name: cloudfoundry
director_uuid: c3119c14-ba8c-4293-8d6e-83f76d94cd46 # get this from bosh status
releases:
- name: appcloud
version: latest
compilation:
workers: 8
network: default
cloud_properties:
ram: 2048
disk: 8096
cpu: 4
update:
canaries: 1
canary_watch_time: 3000-90000
update_watch_time: 3000-90000
max_in_flight: 4
networks:
- name: default
subnets:
- range: 192.168.90.0/24
# Reserved IPs are the IPs that BOSH should not use in the declared range.
reserved:
- 192.168.90.5 - 192.168.90.9
# Static IPs are the IPs that are statically assigned to jobs in this manifest. The BOSH director does not attempt
# to dynamically assign these to new VMs.
static:
- 192.168.90.50 - 192.168.90.75
gateway: 192.168.90.1
# If you configured your BOSH/micro-BOSH to enable DNS, leave the DNS section empty. The BOSH director automatically uses the
# BOSH/micro-BOSH powerDNS IP. If any jobs ever need to resolve DNS entries outside the BOSH powerDNS subdomain
# (*.microbosh by default), configure the powerDNS recursor in your bosh release.
dns:
- 192.168.50.5
cloud_properties:
name: VLAN9
- name: lb
subnets:
- range: 192.168.90.0/24
static:
- 192.168.90.48 - 192.168.90.49
dns:
- 192.168.50.5
cloud_properties:
name: VLAN9
resource_pools:
- name: infrastructure
network: default
size: 28
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 4096
disk: 8192
cpu: 1
env:
bosh:
password: mysecretpw
- name: deas
network: default
size: 1
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 8192
disk: 16384
cpu: 4
env:
bosh:
password: mysecretpw
- name: services_3gb
network: default
size: 2
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 3072
disk: 8192
cpu: 8
env:
bosh:
password: mysecretpw
- name: services_13gb
network: default
size: 1
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 13312
disk: 8192
cpu: 8
env:
bosh:
password: mysecretpw
- name: services_21gb
network: default
size: 2
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 21504
disk: 8192
cpu: 8
env:
bosh:
password: mysecretpw
jobs:
- name: debian_nfs_server
release: appcloud
template: debian_nfs_server
instances: 1
resource_pool: infrastructure
persistent_disk: 8192
networks:
- name: default
static_ips:
- 192.168.90.51
- name: services_nfs
release: appcloud
template: debian_nfs_server
instances: 1
resource_pool: infrastructure
persistent_disk: 10240
properties:
debian_nfs_server:
no_root_squash: true
networks:
- name: default
static_ips:
- 192.168.90.52
- name: syslog_aggregator
release: appcloud
template: syslog_aggregator
instances: 1
resource_pool: infrastructure
persistent_disk: 100000
networks:
- name: default
static_ips:
- 192.168.90.53
- name: nats
release: appcloud
template: nats
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.54
- name: hbase_slave
release: appcloud
template: hbase_slave
instances: 3
resource_pool: infrastructure
persistent_disk: 2048
networks:
- name: default
static_ips:
- 192.168.90.55 - 192.168.90.57
- name: hbase_master
release: appcloud
template: hbase_master
instances: 1
resource_pool: infrastructure
persistent_disk: 2048
networks:
- name: default
static_ips:
- 192.168.90.58
- name: opentsdb
release: appcloud
template: opentsdb
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.59
- name: collector
release: appcloud
template: collector
instances: 1
resource_pool: infrastructure
networks:
- name: default
- name: dashboard
release: appcloud
template: dashboard
instances: 1
resource_pool: infrastructure
networks:
- name: default
- name: ccdb_ng
release: appcloud
template: postgres
instances: 1
resource_pool: infrastructure
persistent_disk: 2048
networks:
- name: default
static_ips:
- 192.168.90.60
properties:
db: ccdb_ng
- name: uaadb
db_scheme: postgresql
release: appcloud
template: postgres
instances: 1
resource_pool: infrastructure
persistent_disk: 2048
networks:
- name: default
static_ips:
- 192.168.90.61
properties:
db: uaadb
- name: vcap_redis
release: appcloud
template: vcap_redis
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.62
- name: services_redis
release: appcloud
template: vcap_redis
instances: 1
resource_pool: infrastructure
persistent_disk: 2048
networks:
- name: default
static_ips:
- 192.168.90.63
properties:
vcap_redis:
port: 3456
password: fksaefblsdf9
maxmemory: 2000000000
persistence:
dir: /var/vcap/store/vcap_redis
- name: uaa
release: appcloud
template: uaa
instances: 1
resource_pool: infrastructure
networks:
- name: default
- name: login
release: appcloud
template: login
instances: 1
resource_pool: infrastructure
networks:
- name: default
- name: cloud_controller_ng
release: appcloud
template: cloud_controller_ng
instances: 1
resource_pool: infrastructure
networks:
- name: default
- name: router
release: appcloud
template: gorouter
instances: 1
resource_pool: infrastructure
networks:
- name: default
default: [dns, gateway]
- name: lb
static_ips:
- 192.168.90.48
- name: health_manager_next
release: appcloud
template: health_manager_next
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.74
properties:
cc_props: ccng
hm_props: health_manager_ccng
- name: dea_next
release: appcloud
template: dea_next
instances: 4
resource_pool: deas
update:
max_in_flight: 8
networks:
- name: default
properties:
dea_next:
stacks:
- lucid64
- name: mysql_node_100
release: appcloud
template: mysql_node_ng
instances: 1
resource_pool: services_3gb
persistent_disk: 12058
properties:
plan: "100"
networks:
- name: default
static_ips:
- 192.168.90.65
- name: mysql_gateway
release: appcloud
template: mysql_gateway
instances: 1
resource_pool: infrastructure
networks:
- name: default
properties:
uaa_client_id: "vmc"
uaa_endpoint: https://uaa.your.domain.org
uaa_client_auth_credentials:
username: kcoleman
password: mysecretpw
- name: mongodb_node_100
release: appcloud
template: mongodb_node_ng
instances: 1
resource_pool: services_21gb
persistent_disk: 63729
properties:
plan: "100"
networks:
- name: default
static_ips:
- 192.168.90.66
- name: mongodb_gateway
release: appcloud
template: mongodb_gateway
instances: 1
resource_pool: infrastructure
networks:
- name: default
properties:
uaa_client_id: "vmc"
uaa_endpoint: https://uaa.your.domain.org
uaa_client_auth_credentials:
username: kcoleman
password: mysecretpw
- name: redis_node_100
release: appcloud
template: redis_node_ng
instances: 1
resource_pool: services_13gb
persistent_disk: 7168
properties:
plan: "100"
networks:
- name: default
static_ips:
- 192.168.90.67
- name: redis_gateway
release: appcloud
template: redis_gateway
instances: 1
resource_pool: infrastructure
networks:
- name: default
properties:
uaa_client_id: "vmc"
uaa_endpoint: https://uaa.your.domain.org
uaa_client_auth_credentials:
username: kcoleman
password: mysecretpw
- name: rabbit_node_100
release: appcloud
template: rabbit_node_ng
instances: 1
resource_pool: services_21gb
persistent_disk: 3072
properties:
plan: "100"
networks:
- name: default
static_ips:
- 192.168.90.68
- name: rabbit_gateway
release: appcloud
template: rabbit_gateway
instances: 1
resource_pool: infrastructure
networks:
- name: default
properties:
uaa_client_id: "vmc"
uaa_endpoint: https://uaa.your.domain.org
uaa_client_auth_credentials:
username: kcoleman
password: mysecretpw
- name: postgresql_node_100
release: appcloud
template: postgresql_node_ng
instances: 1
resource_pool: services_3gb
persistent_disk: 15861
properties:
plan: "100"
networks:
- name: default
static_ips:
- 192.168.90.69
- name: postgresql_gateway
release: appcloud
template: postgresql_gateway
instances: 1
resource_pool: infrastructure
networks:
- name: default
properties:
uaa_client_id: "vmc"
uaa_endpoint: https://uaa.your.domain.org
uaa_client_auth_credentials:
username: kcoleman
password: mysecretpw
- name: backup_manager
release: appcloud
template: backup_manager
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.70
- name: service_utilities
release: appcloud
template: service_utilities
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.71
- name: serialization_data_server
release: appcloud
template: serialization_data_server
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.90.72
properties:
domain: cf.your.domain.org
networks:
apps: default
management: default
nats:
user: nats
password: mysecretpw
address: 192.168.90.54
port: 4222
#required for uaa batch until it is removed
ccdb:
address: 127.0.0.1
port: 12345
roles:
- tag: admin
name: kcoleman
password: mysecretpw
databases:
- tag: cc
name: foobar
ccdb_ng:
address: 192.168.90.60
port: 5524
pool_size: 10
roles:
- tag: admin
name: ccadmin
password: mysecretpw
databases:
- tag: cc
name: appcloud
citext: true
health_manager_ccng:
shadow_mode: disable
cc_partition: ng
uaadb:
address: 192.168.90.61
port: 2544
roles:
- tag: admin
name: root
password: mysecretpw
databases:
- tag: uaa
name: uaa
cc:
srv_api_uri: http://ccng.kendrickcoleman.c0m
ccng:
srv_api_uri: http://ccng.kendrickcoleman.c0m
external_host: ccng
logging_level: debug
bulk_api_password: mysecretpw
uaa_resource_id: cloud_controller
staging_upload_user: upload-user
staging_upload_password: mysecretpw
vcap_redis:
address: 192.168.90.62
port: 5454
password: mysecretpw
maxmemory: 2000000000 # 2GB
router:
status:
port: 8080
user: routeruser
password: mysecretpw
dashboard:
uaa:
client_id: dashboard
client_secret: mysecretpw
users:
- [dash-user, dash-password]
dea:
max_memory: 8192
dea_next:
memory_mb: 8192
memory_overcommit_factor: 4
disk_mb: 16384
disk_overcommit_factor: 4
num_instances: 256
nfs_server:
address: 192.168.90.51
network: 192.168.90.0/24
hbase_master:
address: 192.168.90.58
hbase_master:
port: 60000
webui_port: 60010
heap_size: 1024
hbase_zookeeper:
heap_size: 1024
hadoop_namenode:
port: 9000
opentsdb:
address: 192.168.90.59
port: 4242
hbase_slave:
hbase_regionserver:
port: 60020
heap_size: 1024
addresses:
- 192.168.90.55
- 192.168.90.56
- 192.168.90.57
service_plans:
mysql:
"100":
description: "Shared server, shared VM, 1MB memory, 10MB storage, 10 connections"
free: true
job_management:
high_water: 900
low_water: 100
configuration:
capacity: 500
max_db_size: 10
key_buffer: 512
innodb_buffer_pool_size: 512
max_allowed_packet: 16
thread_cache_size: 128
query_cache_size: 128
max_long_query: 3
max_long_tx: 30
max_clients: 10
max_connections: 1000
table_open_cache: 2000
innodb_tables_per_database: 50
connection_pool_size:
min: 5
max: 10
backup:
enable: true
lifecycle:
enable: true
serialization: enable
snapshot:
quota: 1
warden:
enable: false
postgresql:
"100":
description: "Shared server, shared VM, 1MB memory, 10MB storage, 10 connections"
free: true
job_management:
high_water: 900
low_water: 100
configuration:
capacity: 500
max_db_size: 10
max_long_query: 3
max_long_tx: 10
max_clients: 10
max_connections: 1000
shared_buffers: 320
effective_cache_size: 1142
shmmax: 397410304
checkpoint_segments: 16
checkpoint_segments_max: 50
maintenance_work_mem: 30
backup:
enable: true
lifecycle:
enable: true
serialization: enable
snapshot:
quota: 1
warden:
enable: false
mongodb:
"100":
description: "Dedicated server, shared VM, 250MB storage, 10 connections"
free: true
job_management:
high_water: 230
low_water: 20
configuration:
capacity: 125
max_clients: 10
quota_files: 4
quota_data_size: 240
enable_journaling: true
backup:
enable: true
lifecycle:
enable: true
serialization: enable
snapshot:
quota: 1
redis:
"100":
description: "Dedicated server, shared VM, 20MB memory, 50 connections"
free: true
job_management:
high_water: 410
low_water: 40
configuration:
capacity: 220
max_memory: 20
memory_overhead: 20
max_clients: 50
persistent: true
backup:
enable: true
lifecycle:
enable: true
serialization: enable
snapshot:
quota: 1
rabbit:
"100":
description: "Dedicated server, shared VM, 1MB messages/day, 10 connections"
free: true
job_management:
high_water: 280
low_water: 20
configuration:
capacity: 150
max_disk: 10
max_clients: 10
vm_memory_high_watermark: 0.00587
free_disk_low_water: 0.01832
bandwidth_quotas:
per_day: 1
per_second: 0.01
filesystem_quota: true
serialization_data_server:
upload_token: asdf9asd9asd9ad9ads9ads
use_nginx: true
upload_timeout: 10
port: 8080
upload_file_expire_time: 600
purge_expired_interval: 30
service_backup:
nfs_server:
address: 192.168.90.52
export_dir: /var/vcap/store/shared
service_migration:
nfs_server:
address: 192.168.90.52
export_dir: /var/vcap/store/shared
service_backup_manager:
enable: true
service_snapshot_manager:
enable: true
service_job_manager:
# enable: true
service_lifecycle:
download_url: service-serialization.cf.kendrickcoleman.c0m
mount_point: /var/vcap/service_lifecycle
tmp_dir: /var/vcap/service_lifecycle/tmp_dir
resque:
host: 192.168.90.63
port: 3456
password: mysecretpw
nfs_server:
address: 192.168.90.52
export_dir: /var/vcap/store/shared
serialization_data_server:
- 192.168.90.72
mysql_gateway:
token: uiasdfkasdfophasdfkjadsf
default_plan: "100"
supported_versions: ["5.5"]
version_aliases:
current: "5.5"
cc_api_version: v2
mysql_node:
password: mysecretpw
supported_versions: ["5.5"]
default_version: "5.5"
max_tmp: 1024
redis_gateway:
token: u9sadfy9adfspsadonjasdfhjopadfs
default_plan: "100"
supported_versions: ["2.6"]
version_aliases:
current: "2.6"
cc_api_version: v2
redis_node:
command_rename_prefix: foobar
supported_versions: ["2.6"]
default_version: "2.6"
mongodb_gateway:
token: asdfasdfadfs090uadsf9fsrg
default_plan: "100"
supported_versions: ["2.2"]
version_aliases:
current: "2.2"
cc_api_version: v2
mongodb_node:
supported_versions: ["2.2"]
default_version: "2.2"
max_tmp: 900
rabbit_gateway:
token: asdfuiasdfojiasdfjosadfjks
default_plan: "100"
supported_versions: ["2.8"]
version_aliases:
current: "2.8"
cc_api_version: v2
rabbit_node:
supported_versions: ["2.8"]
default_version: "2.8"
postgresql_gateway:
token: asdfg0uadsfophiasdfhophoads
supported_plan: "100"
supported_versions: ["9.1"]
version_aliases:
current: "9.1"
cc_api_version: v2
postgresql_node:
supported_versions: ["9.1"]
default_version: "9.1"
password: mysecretpw
syslog_aggregator:
address: 192.168.90.53
port: 54321
uaa:
catalina_opts: -Xmx768m -XX:MaxPermSize=256m
url: http://uaa.cf.kendrickcoleman.c0m
resource_id: account_manager
client_secret: somesecret
token_secret: tokensecret
cc:
token_secret: asdfasdfasdfadfs
client_secret: asdfasdfasdffth
admin:
client_secret: erthth45ydbgaerg
login:
client_secret: 457dgnsrgw457sdg
batch:
username: sdfgjwetrysdgf
password: 235dshsbsdgssd
client:
override: true
autoapprove:
- vmc
- my
- portal
- micro
- support-signon
- login
clients:
dashboard:
secret: w4ydbsbJUetdrgdsg
scope: openid,dashboard.user
authorities: uaa.admin,uaa.resource,tokens.read,scim.read,scim.write
authorized-grant-types: client_credentials,authorization_code,refresh_token
portal:
override: true
id: portal
scope: scim.write,scim.read,openid,cloud_controller.read,cloud_controller.write
authorities: scim.write,scim.read,openid,cloud_controller.read,cloud_controller.write,password.write,uaa.admin
secret: portalsecret
authorized-grant-types: authorization_code,client_credentials,password,implicit
access-token-validity: 1209600
refresh-token-validity: 1209600
support-services:
scope: scim.write,scim.read,openid,cloud_controller.read,cloud_controller.write
secret: ssosecretsso
id: support-services
authorized-grant-types: authorization_code,client_credentials
redirect-uri: http://support-signon.cf.kendrickcoleman.c0m
authorities: portal.users.read
access-token-validity: 1209600
refresh-token-validity: 1209600
vmc:
override: true
authorized-grant-types: implicit
authorities: uaa.none
redirect-uri: http://uaa.cloudfoundry.com/redirect/vmc,https://uaa.cloudfoundry.com/redirect/vmc
scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write
scim:
users:
- myemail|mysecretpw|scim.write,scim.read,openid,cloud_controller.admin
- dash-user|dash-password|openid,dashboard.user
To make sure I could deploy this in my homelab, i changed some variables around. In the templates, I reduced the amount of RAM and vCPUs for each VM. As an example, replace the ram and cpu values as seen. Yet, I still didn't have enough horsepower.
- name: services_21gb
network: default
size: 2
stemcell:
name: bosh-vsphere-esxi-ubuntu
version: latest
cloud_properties:
ram: 21504 4096
disk: 8192
cpu: 8 2
env:
bosh:
password: mysecretpw
In addition, I also changed the amount of instances for some of the services
- name: dea_next
release: appcloud
template: dea_next
instances: 4 1
resource_pool: deas
update:
max_in_flight: 8
networks:
- name: default
properties:
dea_next:
stacks:
- lucid64
if your environment is resource constrained like mine, the deployment will fail multiple times. just keep hitting bosh deploy to pick up where it left off.
Now Cloud Foundry should be setup but I didn't have the environment capable of deploying it. If this works for you, please comment below so others will know and I can update it. Thanks
Part 1 - How to install MicroBOSH
Part 2 - Deploy BOSH with MicroBOSH
Part 3 - Deploy Cloud Foundry with BOSH