AWS Spot instance
Concept
Compare to on-demand instance which costs about $0.02/h, spot instance price starts from $0.006, almost a third of the other one. Running a micro instance for a month will be only $4.5.
Why so cheap? One possible reason would be, on-demand instances are charged by hour, which means even if you only used it for one minute then stop your instance immediately, you will still be charged for the full hour.
The only drawback for spot instance is that there is no guarantee for server instance to continue running. When market price goes up, spot instance will be terminated. Try to pick a quiet region if you can, e.g., us-east-1b price history seems very stable 0.006/h for a month, even for past 3 months ( only 2 spikes on Apr. 23)
When instance terminated, all data will be lost. Root ebs will be deleted automatically. If state data is needed, an extra ebs can be attached as user-data storage.There are many different posts showing how to attach another ebs during start-up, like coding in rc.d
http://ec2dream.blogspot.ca/search/label/EBS
http://www.ioncannon.net/system-administration/199/automounting-amazon-ebs-volumes-on-ec2-instances/
but the easiest way is to set it in request, ec2 tool kit only.
- https://forums.aws.amazon.com/thread.jspa?messageID=200921𱃙
- http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts-spot-instances-applications-ebs.html
- http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RequestSpotInstances.html
ec2-request-spot-instances -p 0.006 -t t1.micro -n 1 -b ‘/dev/sdf=snap-a023dfdf::false’ ami-746fc91d -z us-east-1b -k mykeypaire -g my-group -r persistent
- Can not use sda1 (root), otherwise error: snapshotId cannot be modified on root device
- /dev/sdx start from a letter >f, in the example it uses b, I encountered an error saying snap id can only be changed on ebs device.
- Don’t forget the key-value pair and security group. They are key to access.
- -r persistent means repeat request.
Setup OS (environment, tools, your app, etc)
Spot instance should start working for your order immediately after request fulfilled, the perfect solution would be start from your private AMI.
Create AMI
To create an AMI is to request a on-demand instance, setup everything you need, then
- Option A: created from an existing instance, better to stop the instance before
- Option B: create an ebs snapshot, from which create a new image.
Write down your image id. then request spot instance. Request can be done through AWS console or ec2 tool kit.
Setup ec2 tool kit
Download zip file from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
Simply following http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/setting-up-your-tools.html
export EC2_HOME=~/tools/ec2…
export JAVA_HOME=/usr
Generate X509 cert following http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-credentials.html#using-credentials-certificate
upload those 2 files to your work machine under ~/.ec2 folder
$ export EC2_PRIVATE_KEY=~/.ec2/pk-blahblah.pem
$ export EC2_CERT=~/.ec2/cert-blahblah.pem
Test setup using a command like ec2-describe-regions
It should work.
Those export and path setup can be put into ~/.bash_profile or ~/.bashrc
Manually attach ebs
Created a new ebs 1G
Attach to new instance manually in AWS console
device name will change to /dev/xvdf from /dev/sdf in ubuntu
ebs needs format before mount
sudo mkfs -t ext3 /dev/xvdf
sudo mount -t ext4 /dev/xvdf /home/ubuntu/folder_to_mount
ref: http://yoodey.com/how-attach-and-mount-ebs-volume-ec2-instance-ubuntu-1010
Set trust scp/ssh for rc2 instance on your own VPS
local: ssh-keygen -t dsa -f $HOME/.ssh/id_dsa -P ”
remote:(~/.ssh) cat id_dsa >> authorized_keys
ref: http://www.csua.berkeley.edu/~ranga/notes/ssh_nopass.html
Bad news, spot instance is not included in free tier.
All the saving cost export I spent so far seems useless, and I will revisit this post when my 12 month free tier expired.
Push notification coding memo
Push messages are all from apple push server, one for each environment: sandbox and production.
You need to build a push client to send message to Apple push server in PHP or ruby. Push client needs ssl certificate from apple. To get it, enable push feature in your AppId, import cert, then create new provision file.
Considering Xtify or UrbanAirship if you or your clients don’t have your/their own VPS. Both Xtify and UrbanAirship offer free tier membership, 1 million push messages per month.
Documentation from Xtify and UrbanAirship are a little bit different, Xtify needs you export private key under APN certificate item as push certificate, while UrbanAirship document stated that you need to export the APN certificate itself (not the private key under), otherwise when exporting certificate, it will report error saying you are importing a development certificate.
Xtify doesn’t need credit card to register, UrbanAirship won’t allow you create App for production until you give away your credit card number.
Xtify has a magic schedule settings, forgetting set this will cause not sending message, ensure to activate it and check measurement report before you revoke your certificate and start over.
Support response from UrbanAirship is mush faster, my first question was answered in 3 hours, amazing.
Call Apple feedback service frequently to eliminate inactive device from your send out list.
Revisit Date.parse, time zone, dst, and sqlite issues
I thought Date.parse() was very easy, until some time zone and DST (daylight saving time) issues came to me today.
First problem, missing time zone means UTC
DateTime.parse(“2012-04-19 12:12:12″) is default to UTC, I need find out the correct time zone. OK, MDT for now. But, what if DST finished?
That means I can’t hard code MDT in date string.
Need to find out a way to check if date_to_parse is in DST period or not.
Assuming the machine in which the code is running on is at the same time zone as the date_to_parse.
date_s = "2012-04-19 12:12:12"
date = DateTime.parse("#{date_s}")
time_zone = Time.local(date.year, date.month, date.day).isdst ? "MDT" : "MST"
puts DateTime.parse("#{date_s} #{time_zone}")
OK, it works very well. I don’t have to worry about to change the time zone back when Winter is coming.
Second issue, date in sqlite3.
Sqlite3 doesn’ t have a datetime internal date type. It convert it to string or integer.
So when saving date into sqlite3, right way should be either convert date to integer (unix epoch),
Time.parse(date_s).to_i
or convert it to string WITH TIME ZONE.
date.strftime("%Y-%m-%d %H:%M:%S %Z")
Third one, comparing date in sqlite3
Remember no date_diff in sqlite3, date is either string (not compare friendly) or integer.
select date, strftime('%s',date) as orig_date,strftime('%s','#{occurrence.date.strftime("%Y-%m-%d %H:%M:%S %Z")}') as new_date from occurrence
where abs(orig_date - new_date) < 3600 * 10
SSL3_GET_SERVER_CERTIFICATE error in JJ’s VPS manager
If you like me own a busy VPS at dreamhost, you probably already falling in love with JJ’s VPS manager, a tool to dynamically adjust server memory according server load.
Everything works OK until today I got SSL3_GET_SERVER_CERTIFICATE error.
CURL encountered an error. ‘SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed’
Because it happens on my two different VPSs, I doubt it’s Dreamhost API server problem. Google results turn out there is a way to ignore server certificate.
Quick workaround:
- vi ~jj/libs/DH_API/dreamhost.api.php
- find curl_setopt
- add the following 2 lines
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
Restart daemon, JJ seems happy now.
I know it’s not safe, if you know what’s happening on Dreamhost API server, please notify me to add those check back.
Convert Validator from EVil to DataAnnotation
We were using Evil Validation library for a while, works OK, but doesn’t support WPF/Silverlight and MVC very well. Fortunately, it’s not too hard to convert it to DataAnnotations.
Evil code example:
class EmailAddress
{
public EmailAddress(string value)
{
Value = value;
}
[ValidateRegex(@"\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,3}", "Invalid recipient email address")]
public string Value { get; set; }
}
public static bool ValidateEmailAddress(string emailAddress)
{
return ((new EmailAddress(emailAddress)).IsValid());
}
To DataAnnotation:
class EmailAddress
{
public EmailAddress(string value)
{
Value = value;
}
[RegularExpression(@"\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,3}", ErrorMessage = "Invalid recipient email address")]
public string Value { get; set; }
}
public static bool ValidateEmailAddress(string emailAddress)
{
EmailAddress emailToValidate = new EmailAddress(emailAddress);
// Watchout the last bool flag, indicate whether validate all properties. (What else? just modified one?)
return Validator.TryValidateObject(emailToValidate, new ValidationContext(emailToValidate, null, null), null, true);
// For one property model, we can also use this syntax.
ValidationContext validationContext = new ValidationContext(emailToValidate, null, null);
validationContext.MemberName = "Value";
return Validator.TryValidateProperty(emailAddress,
validationContext
, null);
}
Invalid Product Id?
I’ve re-visited this same blog post again and again at least 10 times since I started iOS developing, every time with different answer, to save me search time in the future, I quoted the list here and added my 2 cents for how to solve invalid-product-id problem.
List from Troy:
- Have you enabled In-App Purchases for your App ID?
- Have you checked Cleared for Sale for your product?
- Have you submitted (and optionally rejected) your application binary? (I don’t think this is necessary, in fact, after you submit, you can add new product into the app anymore)
- Does your project’s .plist Bundle ID match your App ID? (This is what I got today, App ID and bundle ID were created by another guy)
- Have you generated and installed a new provisioning profile for the new App ID? (got me 2 times)
- Have you configured your project to code sign using this new provisioning profile? (got me 1 time)
- Are you building for iPhone OS 3.0 or above?
- Are you using the full product ID when when making an
SKProductRequest? (got me 1 time) - Have you waited several hours since adding your product to iTunes Connect? (got me 1 time)
- Are your bank details active on iTunes Connect? (via Mark)
- Have you tried deleting the app from your device and reinstalling? (via Hector, S3B, Alex O, Joe, and Alberto) (This solved my problem 3 times)
- Is your device jailbroken? If so, you need to revert the jailbreak for IAP to work. (via oh my god, Roman, and xfze)
My suggestions:
- Test listing product from simulator first, then move on to device. Ensure product id is OK.
- Uninstall can fix the problem happened on device most of time.
- Delay on new created product happens, at least to me 2 times, but not always, check your local settings carefully following the list above.
- Once old product id being removed, the new product id still can not conflict with the old id, it’s a soft delete in Apple’s system.
- Don’t bother auto-renewable subscription product, unless your are building a newsstand/magazine app.
- Don’t be too greedy on consumable in-app purchase product, start small, you can add big one in later version. Apple is very strict on those coin-like products.
Build an iOS app access only Drupal CMS site
Drupal is based on mysql, in the new version 7 it gives option to choose sqlite. Either way, connecting from iOS app directly to Drupal db is a problem.
Somebody posted mysql library for mac OS, even for iSO, but it’s not public API. Apple won’t like it.
Solution: RESTful service on Drupal
Installation process is painful, because the missing sypc.php can’t be done through Update management web interface module.
Turn on RESTful service, getting node/1 is not a problem, but I couldn’t make user/1 to work, later I realized it’s permission problem, had to grant ‘pull user profile’ right to anonymous user.
The official doc stated that RESTful comes with basic authentication sub module, but I couldn’t find it, the only one available in RESTful config panel is Session Authentication, which means admin must log in to be able to call user/1
Tried http://username:passwrod@mysite/test/user/1, and using code to pass credential, like this code in ruby, no good.
uri = URI('http://test.site.com/drupal/rest/user/1.json')
req = Net::HTTP::Get.new(uri.request_uri)
req.basic_auth 'user', 'psw'
res = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(req)
}
p res
puts res.body
It seems Drupal RESTful service is having problem in basic authentication. Or I just too dumb to figure it out.
My project doesn’t have a requirement to protect web content, which means those nodes should be only available to iOS client.
Work Around: Secure site module
One problem to fix is to apply patch for non-Apache-module mode php site, mine is CGI, adding
RewriteRule .* – [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
to .htaccess, tah-dah, it works!
While the database setup in secure site module is still too messy to me, I just end up with setup a highly secured anonymous user for now. Anyway, content has been protected.
About testing Drual REST servcie
GET is fine, but http://test.site.com/drupal/rest/taxonomy_term/selectNodes is a POST only URI.
Test method 1: download Poster Firfox add-on, set Content Type to application/x-www-form-urlencoded, parameter to tid=1
To get json result, set headers to application/json.
Test method 2:
curl http://test.site.com/drupal/rest/taxonomy_term/selectNodes –form ‘tid=1′ -u user:pwd –header ‘Accept:application/xml’
Watch out, by default form-urlencoded is not checked after installation.
Passing json as request data is much easier.
curl http://test.site.com/drupal/rest/taxonomy_term/selectNodes –d {‘tid’:1} -u user:pwd –header ‘Accept:application/json’
doesn’t work for me, don’t know what missing.
Back to Session Authentication
According to this comment, to properly setup a session in RESTfule service is this:
- Get session id through an empty body POST to rest/system/connect, parse response to get session id. (Must set req['Accept'] = ‘application/json’ or xml if using ruby)
- POST to rest/user/login with parms like sessid=2c038ba6326f12f132502e98434857cc&username=<username>&password=<password>
- Parse body.
This works, but how to keeping session in the following rest request is not clear, more specific, how to pass sess id in request?
Here is another way based on Michael Cole’s comment, calling login directly, his example is saving session name/id in a cookie file, then passing it as session cookie in the following request. If actually can be passed as Cookie attribute in header.
Here is my explanation in ruby code,
def login(username, psw)
uri = URI("http://test.mysite.com/drupal/rest/u/login.json")
req = Net::HTTP::Post.new(uri.request_uri)
req.content_type = 'application/x-www-form-urlencoded'
req.set_form_data("username" => username, 'password' => psw)
#req['Accept'] = 'application/json' #another style if url not end with .json
res = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(req)
}
if Net::HTTPOK
json = JSON(res.body)
@session_id = json["sessid"]
@session_name = json["session_name"]
end
end
def get_post(id)
uri = URI("http://test.mysite.com/drupal/rest/node/#{id}.json")
req = Net::HTTP::Get.new(uri.request_uri)
req["Cookie"] = "#{@session_name}=#{@session_id}"
res = Net::HTTP.start(uri.host, uri.port) {|http|
http.request(req)
}
if Net::HTTPOK
json = JSON(res.body)
return json["body"]["und"][0]["safe_value"]
end
end
Best doc I found on drupal REST service.









