Friday, December 26, 2014

Why I will liquidiate and/or transfer all funds from E-trade January 1st

I have often extolled the virtues of E-Trade to people. Today things have changed. I LOVED that I could have checking, savings, ForEx, IRA, and RothIRA in one place. The only limitation was I could not have a self directed IRA, but that was minor. VERY few IRA's offer this feature, so no trouble here. I have, however here found, that I can cite in pure form 4+ examples of E-trade representatives LYING TO ME IN WRITING!

I actually watched in the late 90's when E-trade, like many folks tried to diversify into banking. I understood why - Bank Deposits are cheap relative to the rest. They give margin loans, and nothing could be cheaper then bank deposits where they pay very little. Note, today they pay roughly 0.02% but charge 8.4% under $50K of margin debit (and have done so on my first disputed credit card transaction in 10 years).

It's been at least 10 years. I don't know quite how long, but I've had an account with them FOREVER (in my mind). I had a debit card stolen, and they took care of me once when my account was hacked (not my fault - they called me) - both within days. I was happy.

I was a bit frustrated when time after time they denied charges while I was abroad - this was a common occurrence, but when I called they lifted it right away. They did leave messages saying "your card is being used abroad," which I always got upon a return to the US, but a pain in the butt. Still letting it go.

But I need to share recent behavior with them, and why you should NEVER trust them:

1) Hotels will pillage your money - this is common practice everywhere, but still a pain in the butt. When I slide my card they take the money RIGHT AWAY!. When you check out of a hotel, they will HOLD THE MONEY FOR AS LONG AS 2 WEEKS! Now in their defense, this is a common problem across all issuers of debit cards so I even let them go on this one. It sucks. I don't like, nor have a credit card any more, but this was NEVER a problem back them.

2) Here we go....FIRST TIME IN 10+ YEARS I CONTEST A CHARGE. I reserved a room at the Omni in Austin for SXSW. When I realized I wasn't going, I cancelled my flight on Southwest....no problem. I also cancelled my hotel reservation with the Omni. Across various stays, I and my co-travelers have spent easily $20K at Omni hotels in the last 10 years. I still got a charge for $450 on my card.

I of course first called the Omni and let them know. I called several times. They kept saying they were "looking into it" but never responded to me once. Bummer - Luckily I used a credit card so I am safe right? I spent 3 weeks working with the hotel, to no avail. I will now share with you a transcript (with comments) of my last 3 months. And this is why I saw STAY AWAY FROM ETRADE!


10-20: I give up on working with Omni and call E-Trade. I send them a secure message saying:
This transaction:
OMNI AUSTIN DOWNTO,AUSTIN TX REFID:464275682054572;
Is not valid. I made a reservation with them months ago, but cancelled it several weeks before the stay, well within the bounds of the terms. I contacted them numerous times to remove the charge, and they have not been helpful.
While I've had nothing but good service/results from Omni til now, I need to contest this charge. Please let me know what I need to do. If I need to fill out paper forms please send to thomasgal@gmail.com or fax to:
XXX-XXX-XXXX
-Tom


11-14: I send another note
Sent: 11/14/2014 12:25 PM
Reference Number: 3514379

Customer Message: 

I have been laboriously trying to fix an unauthorized charge on my card.

I cancelled a hotel reservation months ago now, got the unauthorized charge a month ago, called the hotel to fix it and didn't get a response, just 'we'll get back to you.' I sent in my dispute form a week ago and it looks like you received it Tuesday. 

Can I get an update here? I have not done this before, but the agent told me I would receive a crpedit rapidly. Perhaps the bank holiday caused some delays, but can you please just advise on the process? Thanks for your help

I get a response and the full transaction is included:

Sent: 11/15/2014 10:36 PM
Reference Number: 3514379

Customer Service Message:


Dear THOMAS GAL,

Our records indicate that you have been contacted via phone and/or Secure Message regarding message # 3514379 submitted to E*TRADE FINANCIAL on 11/14/2014 12:25 PM.

If you feel that your message was not properly addressed or you need further assistance, please reply to this message or contact customer service at 1-800-ETRADE-1 (1-800-387-2331) from 7 a.m. to midnight ET.

Sincerely,

E*TRADE Customer Service
Sent: 11/15/2014 01:51 PM
Reference Number: 3514379

Customer Message: 

Thank you!


Apparently at this point their computer messed up and forgot about me so I emailed them back. Here was the response.

Sent: 11/15/2014 11:40 AM
Reference Number: 3514379

Customer Service Message:


Dear Thomas Gal,

Thank you for your message regarding the process of receiving a reimbursement for an unauthorized charge.

I have connected the document you sent in to the pending case number that is open regarding this matter. Normally, this happens automatically but for some reason it did not immediately continue the open case. I will follow up with this matter on Monday to ensure that everything is processing correctly. At this point, the processing of the document you submitted should be processed within 2-3 business days at which point you should receive a temporary credit while this matter is investigated. If there is a change to this process or time frame I will contact you by secure message. I apologize for the delay in processing experienced in this matter and any confusion or inconvenience this may have caused.

I hope this information is helpful. I want you to know that I value and appreciate your business. Should you have additional questions regarding this matter or any other, please feel free to respond to this message or contact our Customer Service team at 1-800-387-2331 (International +1 678-624-6210). Our representatives are available 24 hours a day, seven days a week.

Sincerely,

Bryce Sagers
Mon-Fri 8:30am-5:00pm MST
1-800-ETRADE-1 (1-800-387-2331)
Elite Sr. Financial Services Representative
E*TRADE Securities LLC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WRITTEN LIE 1. 2-3 DAYS
WRITTEN LIE 2. TRY YOUR MERCHANT FIRST!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sent: 11/17/2014 05:15 PM
Reference Number: 3517064

Customer Message: 

Thanks. I tried to resolve with them first, and didn't get any responses.

I'll be patient. I just have not gone through this before so I wasn't aware of the process.

Sent: 11/17/2014 02:27 PM
Reference Number: 3517064

Customer Service Message:


Dear Thomas Gal,

This message is regarding your transaction/merchant dispute claim.

I am writing to follow up with some more information I have regarding this matter. Your dispute form is in process but a provisional credit is not available in this type of claim. Provisional credits are provided for fraudulent charge claims but not for merchant disputes. The best course of action in these cases, if you have not already done so, is to try and resolve the matter with the vendor that charged your card. Usually, this results in the quickest possible reimbursement of funds. Either way, what our claims department will do is contact the merchant/vendor and try to resolve the issue based on the information you have provided and request that they reverse the charge. If this request is successful, you should see a credit in your account in 7-10 business days. Please allow 3-5 days for this request to be completed. I will follow up with you by the end of this week regarding the status of this matter via secure message.

I hope this information is helpful. I want you to know that I value and appreciate your business. Should you have additional questions regarding this matter or any other, please feel free to respond to this message or contact our Customer Service team at 1-800-387-2331 (International +1 678-624-6210). Our representatives are available 24 hours a day, seven days a week.

Sincerely,

Bryce Sagers
Mon-Fri 8:30am-5:00pm MST
1-800-ETRADE-1 (1-800-387-2331)
Elite Sr. Financial Services Representative
E*TRADE Securities LLC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WRITTEN LIE 3 7-10 DAYS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your Research: ATM request for account XXXX-6485 has been completed.

To inquire about this service request please send us a secure message and reference Case Number 2291623521096.

Your Research: ATM Request
Status:
Completed
Request Date:
11/18/14
Account Number:
XXXX-6485
Case Number:
2291623521096
Date of Error:
10/06/2014
Amount:
447.35

Sent: 11/20/2014 01:17 PM
Reference Number: 3526230

Customer Service Message:


Dear Thomas Gal,

This message is regarding your debit card unauthorized transaction claim.

I am writing to make you aware that I have received confirmation that merchant dispute department did receive your original form and the matter has been under investigation since 11/12/2014. That being said, it appears as if this type of investigation may take longer than I previously mentioned. I will continue to monitor this matter for you and send another update next week unless a resolution comes earlier.

I want you to know that I value and appreciate your business. Should you have additional questions regarding this matter or any other, please feel free to respond to this message or contact our Customer Service team at 1-800-387-2331 (International +1 678-624-6210). Our representatives are available 24 hours a day, seven days a week.

Sincerely,

Bryce Sagers
Mon-Fri 8:30am-5:00pm MST
1-800-ETRADE-1 (1-800-387-2331)
Elite Sr. Financial Services Representative
E*TRADE Securities LLC



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WRITTEN LIE 4. 14-21 days (approx)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sent: 11/25/2014 11:42 AM
Reference Number: 3533389

Customer Service Message:


Dear Thomas Gal,

This message is regarding your atm charge merchant dispute claim.

I am writing to make you aware that this situation is still under review. I have received notification that the timeframe form merchant dispute resolutions can take up to 45 days after receiving the necessary paperwork has been received. You sent in the paperwork on 11/11/2014 which means that the resolution of this matter may not be reached until 45 days after that point. Unfortunately, I will not be able to continue monitoring this situation as I am being transferred to another department. Please keep an eye on any account notices you may receive between now and the end of December and feel free to contact us at the phone number below or by secure message if you have any additional questions or need any clarification of the process.

I hope this information is helpful. I want you to know that I value and appreciate your business. Should you have additional questions regarding this matter or any other, please feel free to respond to this message or contact our Customer Service team at 1-800-387-2331 (International +1 678-624-6210). Our representatives are available 24 hours a day, seven days a week.

Sincerely,

Bryce Sagers
Mon-Fri 8:30am-5:00pm MST
1-800-ETRADE-1 (1-800-387-2331)
Elite Sr. Financial Services Representative
E*TRADE Securities LLC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WRITTEN LIE 5 45 days!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 


Sent: 11/25/2014 05:48 PM
Reference Number: 3533389

Customer Message:

While this process may take a long time for you, I have reported an unauthorized charge originally 10/21 with you including the exact description of the charge and charge reference number (message ID 3470104), and much earlier with the vendor. I have been very diligent, and Federal law says I am not liable for unlawful charges on my card. It's been well over 45 days since I was subjected to this, and it's fine if you don't want to count the time I spent with the vendor, but then you shouldn't ask me to do that first.

I am at the point of reporting this to an authority because while you have been very polite, I have gotten zero out of this process in actual results, and I'm due a credit long ago.
Sent: 11/26/2014 04:30 PM
Reference Number: 3533389

Customer Service Message:


Dear Mr. THOMAS S GAL,

Thank you for your message regarding your account ending in 6485.

Your merchant dispute 2291623521096 is currently under investigation and it may take up to 45 days. Since is a non Reg-E merchant dispute, unfortunately you will not receive provisional credit during the investigation.

Should you have additional questions, please feel free to contact us via secure message or phone. We are available 7 days a week 24 hours a day at 1-800-387-2331 or internationally at +1-678-624-6210.

Sincerely,

Carmichael Kimoto
Mon-Fri 9AM - 5:30PM ET
1-800-ETRADE-1 (1-800-387-2331)
Financial Services Representative
E*TRADE Securities LLC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
THE LIES KEEP ON COMING! NEVER TRUST ETRADE!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

HERE WE KEEP GOING!


So Phase 3 comes...

Sent: 12/06/2014 08:23 PM
Reference Number: 3533389

Customer Message: 

Pleas advise. We're past 45 days since my initial report, and I haven't heard anything in 2 weeks. 

You have made a clear case for not trying to resolve anything with the merchant if they don't fix things on the first day. The time I spent contacting them was all a waste. I've been out $500 for 2 months :(
Sent: 12/07/2014 12:30 AM
Reference Number: 3533389

Customer Service Message:


Dear Thomas Gal,

Thank you for your message regarding account update.

Unfortunately I will not be able to provide you with any update about the dispute but you will get the dispute outcome by an official letter sent to you once the investigation process is complete. I understand it is a long process and it may look like taking forever. Please feel free to contact us if you do have any more questions. E*TRADE appreciate your business.

Sincerely,

Dennis Otieno
Mon 09:00-800EST, Thr & Fri-2:00-1:00am, Sat 1:30p
1-800-ETRADE-1 (1-800-387-2331)
Financial Services Representative
E*TRADE Securities LLC

Sent: 12/19/2014 01:37 AM
Reference Number: 3533389

Customer Message: 

We're now pushing well up to 60 days. What is going on here?
Sent: 12/07/2014 12:30 AM
Reference Number: 3533389


Sent: 12/20/2014 10:18 AM
Reference Number: 3533389

Customer Service Message:


Dear Mr. Gal,

Thank you for your message regarding transaction issue.

I've checked into your request and the status is still pending. I understand your frustration at the moment and continuing patience; the investigation is still ongoing.

Should you have any further questions, or if you require additional assistance or clarification, we are available 24/7. Please feel free to send us a secure message, or contact us by phone, at 1-800-387-2331.

Sincerely,

Jason Akpolo
Mon -- Fri 8:30 AM ET -- 5:00 PM ET
1-800-ETRADE-1 (1-800-387-2331)
Senior Financial Services Representative
E*TRADE Securities LLC
Sent: 12/20/2014 07:40 PM
Reference Number: 3533389

Customer Message: 

How can you understand my frustration? That transaction happened almost 3 months ago. I reported it to you on 10/21, and you guys keep telling me it takes 45 days, and we're way passed that. You guys keep LYING to me, and brushing off. This is NOT customer service - It's BS.... I'm filing a complaint with the CFPB. If I find an unauthorized charge on my account, I'm not required to pay it, but you guys have happily collected my money and left me hanging. You need to issue me a credit back. I've had an account with you for a decade, and charged $100s of thousands of dollars and never contested a charge once. This is not the behavior of an institution that has my back, but one taking advantage.
Sent: 12/20/2014 10:18 AM
Reference Number: 3533389

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LET THE RUN AROUND CONTINUE!!!!!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sent: 12/22/2014 02:45 PM
Reference Number: 3569892

Customer Message: 

What does this mean?

Mon Dec 22 01:36:51 2014 Electronic Transaction Request Cancelled

We have cancelled your request for the following Electronic Funds Transfer Dispute: 

Account: INDIVIDUAL - 6485
Case number: 2291214729096

No information provided.....
Sent: 12/22/2014 11:00 PM
Reference Number: 3569892

Customer Service Message:


Dear Mr. Gal,

Thank you for your message regarding the status of your ATM dispute request for your E*TRADE Brokerage account xxxx-6485.

I do apologize for the concern that this has caused you. An inquiry has been submitted to our Debit Card department to check why 2291214729096 was cancelled, and to check the status of your dispute request. Once I receive the results of the inquiry, I will send you a secure message with the information.

Please feel free to contact us by sending a secure message or by calling 1800-3872331 for any question or concern regarding your account. Customer Service is open 24 hours.

Sincerely,

Rose Marie B. Quiambao
Mon - Fri, 4:00 p.m. - 1:00 a.m. ET
1-800-ETRADE-1 (1-800-387-2331)
Financial Services Representative II
E*TRADE Securities, LLC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Can you guys smell the same BULLSHIT I CAN?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sent: 12/23/2014 10:11 PM
Reference Number: 3569892

Customer Service Message:


Dear THOMAS GAL,

Our records indicate that you have been contacted via phone and/or Secure Message regarding message # 3569892 submitted to E*TRADE FINANCIAL on 12/22/2014 02:45 PM.

If you feel that your message was not properly addressed or you need further assistance, please reply to this message or contact customer service at 1-800-ETRADE-1 (1-800-387-2331) from 7 a.m. to midnight ET.

Sincerely,

E*TRADE Customer Service

Sent: 12/24/2014 12:03 AM
Reference Number: 3572423

Customer Service Message:


Dear Mr. Gal,

This message is regarding your ATM Dispute inquiry for your E*TRADE Brokerage account xxxx-6485.

I have received the results of the inquiry from our Debit Card department. The research request 2291214729096 was automatically cancelled after 60 days. Please note that this cancellation did not affect the ongoing investigation. The case is still currently under investigation as the dispute forms were received on 11/11/2014. The investigation will be completed after 45 days. Once this is completed, you will receive the final credit in your E*TRADE account.

Please feel free to contact us by sending a secure message or by calling 1800-3872331 for any question or concern regarding your account. Customer Service is open 24 hours.

Sincerely,

Rose Marie B. Quiambao
Mon - Fri, 4:00 p.m. - 1:00 a.m. ET
1-800-ETRADE-1 (1-800-387-2331)
Financial Services Representative II
E*TRADE Securities, LLC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~``
SO JUST TO BE CLEAR:

1st contact = 3-5 days
2nd thru 10th (guess) contact = 45 days
Contact at 45 days = 90 days

Is there anything here but dishonesty, and shadiness? I think not. 


Sent: 12/24/2014 02:12 PM
Reference Number: 3533389

Customer Service Message:


Dear Mr. Gal,

Thank you for your message regarding the merchant dispute in your account XXXX-6485.

I attempted to call you, but unfortunately there was no answer. I am writing with an update for the merchant dispute.

I have followed up with our department that handles the disputes, and I want to apologize for the miscommunication over the last 60 days. To clarify the dispute process may take up to 90 days from the time we receive your dispute paperwork. That being said we have 45 days to verify your claim, and then the merchant has 45 days to validate the charge. This is a total of 90 days. I have confirmed that the process is scheduled for closure on 01/19/2015. 

If the merchant does not provide substantiating information to support their claim your account will be credited those funds back on that date. However if there is additional information brought forth to substantiate their claim, there will not be a credit given. 

Should you have any further questions, or if you require additional assistance or clarification, we are available 24/7. Please feel free to send us a secure message, or contact us by phone, at 1-800-387-2331.

Sincerely,

Daniel Ethington
7:00 AM - 3:30 PM MST Monday - Friday
1-800-ETRADE-1 (1-800-387-2331)
Financial Services Representative
E*TRADE Securities LLC


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WRITTEN LIE 6 (really much more but let's start there) - IT HAS NOW BECOME 90 days. E-trade has a business of lying. Stay away. I have been lied to repeatedly by their staff over 90 days. 

They shut my card off when I travel, they lie to me repeatedly, and I'll be honest:
1) I can't blame an individual
2) It's the institution
3) These are all copy-pastes other than my comments in bold.

Stay away. Go somewhere else. The staff have been amazing, friendly, and patient. They all have, alas, been trained to lie by their corporation. There is no slander or libel here. This is all 100% copied from my transactions with them. 

I will simply wait for Jan 1. Selling now means a lot of taxes now, and transferring money out of my tax free accounts means heavy taxes and penalties April 15. I can delay this 1 year, but will not trust E-trade for a day past!

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A small personal message. EVERY time I talked to the folks at E-trade who answer the phone they were polite, instructive, and patient. If you know me I am anything BUT that. This is a CORPORATE problem, not a problem with those folks. They were real, and just doing what they were told. I would hire them ANYWHERE if you have a quality business, as they will support you. BUT E-trade....your time is done. I'm sorry. Come Jan 1 I will liquidate all my accounts and transfer them elsewhere.

Goodbye!

Friday, November 28, 2014

On Ferguson, Racism, and Police....

My friend, let's call him Mike, posted this recently. What broke my heart was not the end point, but that he posted the picture without a touch of commentary. It is very tough to post something like that to social media and not have it taken in the wrong way. The fact that most of the United States, right now (alas from my perspective, but the protests have shown something), wouldn't batt an eye at this cartoon speaks volumes:



But not a lot of people have really explained why. So I thought it an appropriate time to review some of the basic facts, and really summarize why I believe the situation currently is so infuriating. I'm seriously pissed. I issue here, a serious challenge, to the peopld & government of this United States to fix this. Whether through abolishing grand juries (nobody has them but us, and everyone is still entitled to the same evidentiary hearing in the first place), or allowing referendum for criminal cases. My entire life I have been told that Grand Juries establish if there is evidence for a trial.

1) I'll get to race shortly, but I want to give first an observation which is undeniable:

Not a single witness seems to have disputed that the victim (no matter the cause he is a victim - he was unarmed as we only now know in hind sight) tussled hand to hand with the officer, and shots  were fired.

2) Who doesn't find #1 frightening (I write this from that perspective)? Law of the wild, law of the seas, you call it. We actually would, typically (I believe) give most people in the situation where a gun was tussled over hand to hand, the benefit of doubt. Alas for an officer of the law there is a down side to this. In my book anything premeditated, or remotely intentional is off the table. Involuntary manslaughter? How is that not discussed? Here is the big point.....

3) He is an officer of the law, and he MIGHT have made a mistake. The racism here was not that he was shot and killed. Unfortunately he is an officer of the law, and his JOB is to treat everyone equally according to the law and the constitution. I'm not sure of the proper protocol for handling someone walking down the middle of the street. So unfortunately...

4) The only question, with regard to Officer Wilson is if he made a mistake. Not if he is racist, or intended to murder Brown. If anyone reading this thinks a cop wouldn't be crass to someone in the middle of the street, I challenge you. If you think anyone shouldn't get premeditation thrown out the window, I challenge you. 

I can't blame the Grand Jury either - they are the only ignorant's in this contorted by a corrupted process....

BUT

When I read that the prosecutor has been presenting both sides of the case to the grand jury extensively, I was astounded. Attorneys are required by law to follow certain rules. Judges, if they have a conflict, must recuse themselves. I see this same limitation in the legislative branch with legislators who oversee their donators (i.e. corporations or unions) in any way. The point is, that these district attorneys, judges, and police, all have a very clear relationship, along with the populace that requires drastic measures.

Lawyers are required, by their bar, to represent the interests of their client. How can a prosecutor be expected to behave differently? This prosecutors job was ACTUALLY just to get to a trial. How could none of the judges acknowledge this, and set a precedent, not only for themselves but those around them.

The problem is there is no plaintiff for the people. Nobody but the government can charge someone with a criminal charge. That is why people are taking to the streets. They are shutting down malls, and I wasn't sure if I could get back to San Francisco this afternoon, as there were protests at the West Oakland BART station earlier. 

THIS DID NOT GO TO TRIAL?!?

The race problem here IS NOT WITH OFFICER WILSON. I think this is probably why he is scared. If everyone had not resisted charges, and let a trial happen, I think things would have been much more clear, open, and real. The fact that the prosecutor, in this occasion, seemingly decided to behave in a manner that doesn't befit their duties requires all branches to take action. The fact that there are newspaper articles discussing "if the grand jury proceedings will be made public," are exactly the absurdity that dictates that people take to the streets. 

This all points to Officer Brown as a government, and frankly, media scapegoat, for the underlying truth, that the corruption these days is in the prosecutors office. In this case the prosecutors office failed to dutifully recuse themselves of conflict. These days, cases of non-disclosed, nor explained details (channeling my inner economist) in America are profligate. In fact I've been watching this trend in The Economist, my favorite periodical, for a long time now. So now we all focus on a scared police officer, and what happened in a moment, while millions of minorities are abused in different, yet real ways. 

So the question is how did this not go to trial? Is there an American amongst us, who believes it shouldn't have? By whom, did we believe it should have been tried? It was, by the media it seems? To literally quote the Public Defender's Office in San Francisco:


This ethical failure resulted in the exceedingly rare step of the prosecuting attorney refusing to recommend an indictment against the police officer he was prosecuting.

Everyone tried to brush it under the rug. The police officer had nothing to do with that. He should not be vilified. The fact that he was honest, probably exposed some racists along the way that rejected due process. Let's find them. Not officer Wilson. The prosecutors were not objective, that is clear. Who's job is that to fix by law? The State Bar (Pun enjoyed)? If we disprove those things, then we can have a discussion, but for now, I think we've got it all wrong.

Friday, November 21, 2014

Regular Expressions, Redirects, and Rewrites oh my!

I recently got a request worded as such:
You choose a technical topic related to programming computers that you know a great deal about already, send me the topic so I can read up on it if necessary, and then prepare to give in-depth 10 to 15 minute lecture about the topic to me and I'll ask follow questions.
I thought this would be the perfect thing to share with everyone else as well. One of my biggest strengths is networking, which I had already covered with that person a bit. As well my extensive embedded experience gives me command at the HW-SW interface. We had already touched on optimizing databases and critical code which I view as more abstract computer architecture (things like pipelining and spacial locality effects on cache misses), and things like pull-ups, flash loaders, and bypass capacitors were not an appropriate subject.

I pondered a bit diving into computability, and the things one could evaluate with seminal computing models like the state machine, push down automata, and Turing machines, but this seemed a bit too much to bite off in short order. However thinking about state machines (e.g. a soda machine), gave me an idea that would incorporate a bit of one of my favorite subjects, Regular Expressions (regex), and let me touch on a scenario that we run into often in the modern interconnected web world.

It will also allow me to demonstrate a few interesting things under the hood of the internet. What you often perceive as simply loading a web page is often becoming a server accessing databases and services on the server side to serve a page which may still yet deliver executable code that can load local and cached resources, as well as make browser side calls to further web services. While below I use the example of loading web content this is analogous (via 4 - code, and combinations of other methods) to using any remote procedure call be it REST, SOAP, or some other method which may deliver JSON, XML, or other things instead of an HTML document.

The Problem


I use case 1 as the base problem, and will add to the problem as we come up with decision points down the line:

1) I want to type www.example.com and see the content of www.yahoo.com

The following cases follow as we go through the analysis:

2) I want to be able to share a URL by itself as a way to share the content I am viewing at the moment. (No iFrame)

3) I want query parameters to pass through to my page, easily, without server or browser code (No iFrame)

4) I want cookies to be shareable between domains (no cross domain iFrame, or careful consideration).

5) I want to support https (no matter what you must get certificates).

6) I want a custom path scheme for my content (no CNAME).

7) I want the user to type www.example.com every they want to get to the site (don't send an HTTP 301)

8) I want the user to still see the domain www.example.com even though they see the content from www.yahoo.com (don't send an HTTP 302)

9) The site/content I am integrating with uses a different version of javascript, jquery, node.js, etc. or I need to tightly control the order of javascript execution, for example using a message passing interface (don't use client side javascript without using an iFrame, or heavy focus on synchronization constructs).

Solution 1: HTML iFrame

This is probably the easiest way for most people to think of. HTML has a mechanism for this called an iFrame. This simple document will embed a window with www.example.com in my page. I created a simple page like this at:

http://hl1264.blogspot.com/2014/11/blog-post.html

I will note some limitations:

1) I click a link inside the iFrame. The iFrame content changes, but my URL doesn't. This is ok for me, but say I am on my nth click and wan't to send a link to a friend. When I copy-paste the URL I will go to my blog post, not to the page I wanted to share!

2) Query parameters are not passed through. I actually ran into this problem when I asked someone to forward a domain in a way I'll cover next, but they used an iFrame. While 1) above was an issue, we were using google analytics and while the user might enter:

http://hl1264.blogspot.com/2014/11/blog-post.html?utm_source=Tom&utm_medium=Blogspot&utm_campaign=Example1

I would not get:

www.example.com/?utm_source=Tom&utm_medium=Blogspot&utm_campaign=Example1

which would allow the information to pass through. I would just get a hit on:

www.example.com

So the information was lost. While it's a quick Javascript or PHP script to grab those parameters and append to the iFrame address, I don't want to do this with code because it's messy, and in this particular case the partner would not do this for us as well.

3) Cookies will not automatically be shared without careful consideration. I run into this often when using iFrames to integrate multiple sites. You must carefully navigate this with a smart CNAME, cookie declaration, and possibly certificate which I will give as a real world example at the end.

iFrames are a useful tool, and can be blended with following items in various ways to achieve different end goals. One prime example is blending sites or content that use different versions of javascript or

Solution 2: DNS CName

In DNS parlance a "cname" is short for "canonical name," and means one domain is an alias for another. This is analogous to a file alias in the Mac world, a shortcut in PC land, and a link in the *nix universe. This allows you to say in 1 DNS entry:

www.example.com     CNAME  www.yahoo.com

As long as the site uses relative paths, then you can navigate the entire site and the paths on example.com will mirror those on yahoo.com or whatever target site you choose. For example most sites will have something like this in place:

www.example.com     CNAME  example.com
example.com               A              93.184.216.119

Where the second line is an "A" or "address" record. The above two lines mean that the canonical name of www.example.com is example.com, and the address of example.com is 93.184.216.119.

1) One limitation here is https, which is rapidly becoming the norm for all sites. If you use this method but your domain is different, modern browsers will block the content or warn users that they are likely entering a dicey situation. This is because the domain the user entered will not match the domain on the certificate. There is no way around this, and any valid case will have no trouble justifying the cost of a new certificate for the project.

2) Another limitation is with regard to the path. With this method the path will always mirror that of the aliased site. In the case of www.example.com and example.com this makes obvious sense but there may be cases where you will want a custom mapping whether arbitrary, or for backwards compatibility, but more on that later.

Solution 3: HTTP Protocol

The Hyper Text Transfer Protocol (HTTP) is used to transfer Hyper Text Markup Language (HTML) documents. You get a code 200, for success almost every time you load a web page, and most people will recognize 404 (Not Found - usually mistyped or dead link), and 503 (Service Unavailable - usually server maintinence or overload), and 500 (Internal Server Error - any unhandled exception, for me usually an unhandled PHP error). Within this protocol there is a class of responses (3XX) which are specifically aimed at redirection. 

These codes also fulfill our purpose in some ways:

301 - Moved Permanently

Sending this response would indicate a permanent change. If I clicked on a bookmark, and receive this response, a smart browser should change the URL in my bookmark. Smart search engines should know longer index the old URL, and should index the new URL.

302 - Moved Temporarily OR Found

Moved Temporarily is the old nomenclature, and now Found is used, but this means keep the URL you have. This response is frequently used in parts of sites that are dynamic with respect to site structure, things like AB testing which could guide user subsets to slightly different interfaces. You can still type www.example.com and get your other site content, but by the time you looked up at the address bar you would quickly see the result site, in our case www.yahoo.com so this option is out if you want to see www.example.com as the domain name.

Solution 4 - The code silly!

There are, of course, many ways to do this by writing good old code! Here I present 3 different modalities of using server and browser side code to accomplish the effect. In all cases one can get the path from the request URL and use that in the URL requested, with similar effects to the cname mapping. This is the method we did not consider before in the simple iFrame case. In a simple Pseudocode:

Take www.example.com/path
read www.yahoo.com/path
return the html document received

This is also a simple example analogous to getting data via a SOAP or REST call and applying a CSS style sheet to it

Regex Aside 1

A quick introduction to regular expressions. Most people know that '*' means "everything," and often people will know that 'ap*' will match "ap," "app," "application," "apache" and anything else that starts with "ap." To get particular let's talk about some typical regex syntax ( the particulars which may be platform and language dependant ). Some regex basics:
  • '.' represents any character, except in a character class
    • '.' will match 'a' or 'b' but also 'ab' as there was in fact a character, there just happened to be a second
  • '+' represents one or more of the preceding character
    • a+ will match 'a', 'aa', 'ab' but not 'b'. It will still match 'ba' and 'baa'
  • '*' represents 0 or more of the preceding character - be careful
    • '*' will match everything
    • '.*' will match everything
    • 'ab*a' will match 'aa', 'aba', 'abbbbbba'
  • '?' makes something optional, or indicates 0 or 1 of the preceding character
    • 'ab?a' will match anything with 'aa' or 'aba' in it but not 'abba'
  • () groups a piece of the regex for later reference. The value in the first params can be referred to typically as $1 or \1, the second as $2 and so on....
  • [] are used to indicate a character class, such that multiple characters are possible
    • [aeiou] will match any string with a vowel in it
    • [1234567890] will match anything with a decimal digit in it
  • ^ Matches the start of a string
    • '^ab' matches "abatement"or "absinthe" but not "an abatement system" or "a bottle of absinthe"
  • $ Matches the end of a string
    • 'ing$' matches "fishing", "swimming", and "lounging", but not "stinking fish", "It's to freezing out there  
In PERL, which is similar enough to PHP, Python, and Javascript with respect to regular expressions here is a simple, yet befuddling looking expression that breaks out the pieces of a URL:

if ($uri =~ m!^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?!) {
        print "protocol:$2, domain:$4, path:$5, query:$7, fragment:$9\n";
}

$uri = "http://www.yahoo.com/folder1/page1.html?key1=val1&key2=val2#FragmentOrAnchor\n\n";

gives 

protocol:http, domain:www.yahoo.com, path:/folder1/page1.html, query:key1=val1&key2=val2, fragment:FragmentOrAnchor

$uri = "ftp://ftp.example.com/user1?user=tom&password=12345";

I will leave deconstructing that regex in your favorite scripting language for you as an exercise later. Regex will come in handy again later when we talk about rewrite rules in 5).

Solution 4a: Server Code loads destination site and serves it

In this case code on the server will naively call the content, and can programatically parse the path out and append it to the call. For example in PERL I could write:

$command = "wget http://www.yahoo.com/".$path; # Where # '.' being string concatenation.
print `$command`; #backtick executes the command in the shell and returns stdout

In this case, as with others https is a concern. In this case, however I could insert elements around or inside the page loaded, modify the style sheet, and do many dynamic things. This modality will frequently be used in the form of Server Side RESTful calls.

Solution 4b: SSI

This will normally not be an option as it opens all kinds of security vulnerabilities, but an easy way for demonstration purposes is to use a Server Side Includ (SSI). This command let's the web page execute any command with group permissions that match with apache's permissions. In this case the simple example is:



Adding in the path needs could be accomplished multiple ways via further code, but is messy, and this example is just academic.

Solution 4c: Browser side java script

With java script you can actually accomplish the feat in many ways that parallel some of the other pitfalls. One way to achieve this is to simply change to the destination URL:

var url = "http://www.yahoo.com.com/"

window.location = url;

But this will give you the destination URL in your browser, and really is equivalent to using an HTTP 3XX code to change your destination. Another option is to load the code and re-write the current document with something like the wget above, or even more fun, just insert the iFrame:

document.body.innerHTML= < iframe src = "http://www.example.com" >< / iframe >


The sensible time to do this may be when you are actually receiving for example XML to be rendered according to CSS rules. This is a much more complex but typical case for embedding widgets and 3rd party content into your site. Especially in the case where all of the traffic is between a 3rd party server, this option will reduce latency, but require smart tricks like subdomains to deal with XSS issues. 
The common trouble is, again, different versions of java script and js frameworks. Doing this on the client side also exposes you to race conditions (The 'A' in AJAX is asynchronous), so this route is not recommended in that case either without careful consideration and synchronization.


Solution 5 Rewrite Rules

Solution 5a: Web Server Rewrite Rules

I know that a similar mechanism exists in Windows world, but I'm used to apache, where they have the mod_rewrite module. From the apache documentation:

The mod_rewrite module uses a rule-based rewriting engine, based on a PCRE regular-expression parser, to rewrite requested URLs on the fly. By default, mod_rewrite maps a URL to a filesystem path. However, it can also be used to redirect one URL to another URL, or to invoke an internal proxy fetch.

So you can use this module to map one path to another, as well as parse things like HTTP headers as I mention below. It is regular expression based ( you see regex everywhere ), so you can create logical mappings, not just one to one relationships. Examples aboud on the net but a couple of examples excerpted:

In the example ruleset below we replace /~user by the canonical /u/user and fix a missing trailing slash for /u/user.

RewriteRule   ^/~([^/]+)/?(.*)    /u/$1/$2  [R]
RewriteRule   ^/([uge])/([^/]+)$  /$1/$2/   [R]

The goal of this rule is to force the use of a particular hostname, in preference to other hostnames which may be used to reach the same site. For example, if you wish to force the use of www.example.com instead of example.com, you might use a variant of the following recipe.

# For sites running on a port other than 80
RewriteCond %{HTTP_HOST}   !^www\.example\.com [NC]
RewriteCond %{HTTP_HOST}   !^$
RewriteCond %{SERVER_PORT} !^80$
RewriteRule ^/(.*)         http://www.example.com:%{SERVER_PORT}/$1 [L,R]

# And for a site running on port 80
RewriteCond %{HTTP_HOST}   !^www\.example\.com [NC]
RewriteCond %{HTTP_HOST}   !^$
RewriteRule ^/(.*)         http://www.example.com/$1 [L,R]

Solution 5b: Rewrite Rules with Sub-domains & Cookies

In the modern world, while i can load an iFrame with content from another site, as soon as you run to active content, especially scripts, the domains can become a problem. If I want to run this script:

www.example.com/cgi-bin/script.js

using my cname example, this would be forwarded as a request to:

www.yahoo.com/cgi-bin/script.js. Certificates aside, a modern browser will not let this happen. It's behavior will be somewhere from simply nothing happening (my default behavior for Firefox at the moment), to getting a warning/error about cross domain scripts or security. In addition if I wrote a re-write rule as in example 5) above, based on HTTP cookies

Real World Examples

To be discussed live....

HTTP Refer

Unified CSS and templating for white label applications via CNAME and rewrite rules

Redirect to Mobile site based on User Agent using Apache ModReWrite

Have you noticed that you typically get redirected to an m-dot "m.site.com" site (not ironman site) often on your mobile device? Apache rewrite rules allow access to HTTP headers:

RewriteCond  %{HTTP_USER_AGENT}  (iPhone|Blackberry|Android)
RewriteRule  ^/$                 /homepage.mobile.html  [L]
RewriteRule  ^/$                 /homepage.std.html  [L]

3rd Party Integration example

The complete package. Example will be given in my presentation