CentOS 7 Ngnix 403 Error

Ngnix reports error 403 when access all connects under a directory but works under a different direction named "images". Original I thought it is dir/file permission issue. It turns out it is the default selinux setting. A good read https://www.getpagespeed.com/server-setup/nginx/nginx-selinux-configuration.

Passing parameters into springboot application

• User OS env variable
• Use java -Dk1=v1 -Dk2=v2
• Program argument java -jar my.jar --k1=v1 --k2=v2
• Use an external application.properties file and pointing to it with export spring.config.location=/tmp/application.properties or passing it in with java parameter -Dspring.config.location=/tmp/application.properties

Scala Spark SBT build up fat jar

As of now 2019/05/29, to create a fat jar for your spark project, here are the steps

1). create file under your_project_root/project/assembly.sbt with contents

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")

https://github.com/sbt/sbt-assembly#using-published-plugin

2). add one section in your_project_root/build.sbt
assemblyMergeStrategy in assembly := {
  case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.last
  case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
  case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case "about.html" => MergeStrategy.rename
  case "git.properties" => MergeStrategy.rename
  case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
  case "META-INF/mailcap" => MergeStrategy.last
  case "META-INF/mimetypes.default" => MergeStrategy.last
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

without this section, you'll get errors like:
[error] (assembly) deduplicate: different file contents found in the following:
[error] /home/h0l01if/.ivy2/cache/org.apache.arrow/arrow-vector/jars/arrow-vector-0.8.0.jar:git.properties
[error] /home/h0l01if/.ivy2/cache/org.apache.arrow/arrow-format/jars/arrow-format-0.8.0.jar:git.properties
[error] /home/h0l01if/.ivy2/cache/org.apache.arrow/arrow-memory/jars/arrow-memory-0.8.0.jar:git.properties
[error] deduplicate: different file contents found in the following:

3). run sbt assembly

Git tag and github release

Git Tag

Git tag is nothing but a marker or symbol for a point in time snapshot of your git repository.

There are two types of tags in git:

Lightweight Tag

A pointer to a specific commit. It only has a tag name.

Annotated Tag  

In addition to putting a marker on a specific commit, git physically create the snapshot of all files and keep it internally with a tag message, email, and a date.

To create a tag for a given commit, the command syntax is:

git tag tag_name commit_id

To push the tag to the remote repository

git push remote_name tag_name

Github Release

Release is a concept of github, it internally uses git tag to create a static point of time snapshot. The details can be found at https://help.github.com/articles/creating-releases/.

Releases can be created on:

1. Current head of master or branches

2. Existing tags

3. Recent past commits.

 

 

 

Yahoo Closes Burbank Office

早晨还是阳光灿烂，午后竟然下起了雨。平添了一股阴郁。

回首几近九年，写几句只言片语，是为记。

Null Handling in Hadoop Pig Latin

For chararray type, when you load a dataset, PigStorage will convert empty fields to null. So in any relations, you won't find any empty string but only nulls.

However, in the pig script, if you have a constant as '', it is not treated as null.

So '' is not null return true.
'' is null return not true.

If A is a relation immediately after a load, A.$0 == '' will never be true.

If you compose something manually with GENERATE, it will keep the origin.

B = FOREACH A GENERATE $0, $1, ''; -- Will keep the value as empty string
C = FOREACH A GENERATE $0, $2, (chararry) null; -- Will keep the value as null

Sorting for NULLs

NULL is always treated as smallest value, if you do ORDER BY DESC, it will come last. If you do ASC, it comes first.

hadoop 2.2.0 installation resource

General steps:

http://milinda.pathirage.org/hadoop/yarn/2013/09/29/how-to-setup-multi-node-hadoop-20xyarn-cluster.html#references<br />

It refers to:
http://www.javacodegeeks.com/2013/06/setting-up-apache-hadoop-multi-node-cluster.html
http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
  </property>

should be:
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

How to build 64bit native library or simply download from:
http://shaurong.blogspot.com/2013/11/hadoop-220-centos-64-x64.html

It has a detailed process on how to build from hadoop source. This is the only reliable/working steps I have found so far.
Hadoop build is already moved to maven but the hadoop document still shows old steps with ant (of course it won't work).
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html

Really sucks.

Another error you may have:
$ hadoop fs -ls
ls: `.': No such file or directory

How to solve:
hadoop fs -mkdir $USER

RHEL6 mail attachment with uuencode won't work anymore

The old trick of sending out attachement with
uuencode /tmp/myfile myfile | mail -s "Attachemnt" user1@xyz.nowhere
won't work any more after migrated to RHEL6. All the uuencoded message will display as body contents rather than an attachment.

If you have a script like above, for sure you will get impacted:
http://www.linuxquestions.org/questions/linux-newbie-8/uuencode-issue-with-rhel-6-3-a-4175450188/

Redhat seems to have some kind of workaround but it only opens to its registered users.
https://access.redhat.com/site/solutions/104833

The fix is pretty simply but you'll still need to make changes to your so-far-working-well scripts.
mail -a /tmp/myfile -s "Attachement" user1@xyz.nowhere.

The problem seems coming from the headers the new version introduced.
something like:
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Old version headers just have:

MIME-Version: 1.0

There might be options to disable the new headers, let me know if anybody find them out.